Estimating real-time delay of a video data stream

ABSTRACT

In an arrangement where a physical phenomenon affects a digital video camera and is measured or sensed by a sensor, a delay of a digital video stream from the digital video camera is estimated. The digital video stream is processed by a video processor for producing a signal that represents the changing over time of the effect of the physical phenomenon on the digital video camera. The signal is then compared with the sensor output signal, such as by using cross-correlation or cross-convolution, for estimating the time delay between the compared signals. The estimated time delay may be used for synchronizing when combining additional varied data to the digital video stream for low-error time alignment. The physical phenomenon may be based on mechanical position or motion, such as pitch, yaw, or roll. The time delay estimating may be performed once, upon user control, periodically, or continuously.

TECHNICAL FIELD

This disclosure generally relates to an apparatus and method forreal-time estimating of a delay of a received video data stream from thevideo actual capturing, and in particular synchronization ortime-stamping of a received video data stream, such as video data streamreceived from a video camera in a vehicle, with other real-time signals,such as for overlaying additional data to the video data stream.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Digital photography is described in an article by Robert Berdan(downloaded from ‘canadianphotographer.com’ preceded by ‘www.’)entitled: “Digital Photography Basics for Beginners”, and in a guidepublished on April 2004 by Que Publishing (ISBN—0-7897-3120-7) entitled:“Absolute Beginner's Guide to Digital Photography” authored by JosephCiaglia et al., which are both incorporated in their entirety for allpurposes as if fully set forth herein.

A digital camera 10 shown in FIG. 1 may be a digital still camera whichconverts captured image into an electric signal upon a specific control,or can be a video camera, wherein the conversion between captured imagesto the electronic signal is continuous (e.g., 24 frames per second). Thecamera 10 is preferably a digital camera, wherein the video or stillimages are converted using an electronic image sensor 12. The digitalcamera 10 includes a lens 11 (or few lenses) for focusing the receivedlight centered around an optical axis 8 (referred to herein as aline-of-sight) onto the small semiconductor image sensor 12. The opticalaxis 8 is an imaginary line along which there is some degree ofrotational symmetry in the optical system, and typically passes throughthe center of curvature of the lens 11 and commonly coincides with theaxis of the rotational symmetry of the sensor 12. The image sensor 12commonly includes a panel with a matrix of tiny light-sensitive diodes(photocells), converting the image light to electric charges and then toelectric signals, thus creating a video picture or a still image byrecording the light intensity. Charge-Coupled Devices (CCD) and CMOS(Complementary Metal-Oxide-Semiconductor) are commonly used as thelight-sensitive diodes. Linear or area arrays of light-sensitiveelements may be used, and the light sensitive sensors may supportmonochrome (black & white), color or both. For example, the CCD sensorKAI-2093 Image Sensor 1920 (H)×1080 (V) Interline CCD Image Sensor orKAF-50100 Image Sensor 8176 (H)×6132 (V) Full-Frame CCD Image Sensor canbe used, available from Image Sensor Solutions, Eastman Kodak Company,Rochester, N.Y.

An image processor block 13 receives the analog signal from the imagesensor 12. The Analog Front End (AFE) in the block 13 filters,amplifies, and digitizes the signal, using an analog-to-digital (A/D)converter. The AFE further provides Correlated Double Sampling (CDS),and provides a gain control to accommodate varying illuminationconditions. In the case of a CCD-based sensor 12, a CCD AFE (AnalogFront End) component may be used between the digital image processor 13and the sensor 12. Such an AFE may be based on VSP2560 ‘CCD Analog FrontEnd for Digital Cameras’ available from Texas Instruments Incorporatedof Dallas, Tex., U.S.A. The block 13 further contains a digital imageprocessor, which receives the digital data from the AFE, and processesthis digital representation of the image to handle variousindustry-standards, and to execute various computations and algorithms.Preferably, additional image enhancements may be performed by the block13 such as generating greater pixel density or adjusting color balance,contrast, and luminance. Further, the block 13 may perform other datamanagement functions and processing on the raw digital image data.Commonly, the timing relationship of the vertical/horizontal referencesignals and the pixel clock are also handled in this block. DigitalMedia System-on-Chip device TMS320DM357 available from Texas InstrumentsIncorporated of Dallas, Tex., U.S.A. is an example of a deviceimplementing in a single chip (and associated circuitry) part or all ofthe image processor 13, part or all of a video compressor 14 and part orall of a transceiver 15. In addition to a lens or lens system, colorfilters may be placed between the imaging optics and the photosensorarray 12 to achieve desired color manipulation.

The processing block 13 converts the raw data received from thephotosensor array 12 (which can be any internal camera format, includingbefore or after Bayer translation) into a color-corrected image in astandard image file format. The camera 10 further comprises a connector19, and a transmitter or a transceiver 15 is disposed between theconnector 19 and the image processor 13. The transceiver 15 may furtherincludes isolation magnetic components (e.g. transformer-based),balancing, surge protection, and other suitable components required forproviding a proper and standard interface via the connector 19. In thecase of connecting to a wired medium, the connector 19 further containsprotection circuitry for accommodating transients, over-voltage andlightning, and any other protection means for reducing or eliminatingthe damage from an unwanted signal over the wired medium. A band passfilter may also be used for passing only the required communicationsignals, and rejecting or stopping other signals in the described path.A transformer may be used for isolating and reducing common-modeinterferences. Further a wiring driver and wiring receivers may be usedin order to transmit and receive the appropriate level of signal to andfrom the wired medium. An equalizer may also be used in order tocompensate for any frequency dependent characteristics of the wiredmedium.

Other image processing functions performed by the image processor 13 mayinclude adjusting color balance, gamma and luminance, filtering patternnoise, filtering noise using Wiener filter, changing zoom factors,recropping, applying enhancement filters, applying smoothing filters,applying subject-dependent filters, and applying coordinatetransformations. Other enhancements in the image data may includeapplying mathematical algorithms to generate greater pixel density oradjusting color balance, contrast and/or luminance.

The image processing may further include an algorithm for motiondetection by comparing the current image with a reference image andcounting the number of different pixels, where the image sensor 12 orthe digital camera 10 are assumed to be in a fixed location and thusassumed to capture the same image. Since images are naturally differ dueto factors such as varying lighting, camera flicker, and CCD darkcurrents, pre-processing is useful to reduce the number of falsepositive alarms. Algorithms that are more complex are necessary todetect motion when the camera itself is moving, or when the motion of aspecific object must be detected in a field containing other movementthat can be ignored. Further, the video or image processing may use, orbe based on, the algorithms and techniques disclosed in the bookentitled: “Handbook of Image & Video Processing”, edited by Al Bovik, byAcademic Press, ISBN: 0-12-119790-5, which is incorporated in itsentirety for all purposes as if fully set forth herein.

A controller 18, located within the camera device or module 10, may bebased on a discrete logic or an integrated device, such as a processor,microprocessor or microcomputer, and may include a general-purposedevice or may be a special purpose processing device, such as an ASIC,PAL, PLA, PLD, Field Programmable Gate Array (FPGA), Gate Array, orother customized or programmable device. In the case of a programmabledevice as well as in other implementations, a memory is required. Thecontroller 18 commonly includes a memory that may include a static RAM(random Access Memory), dynamic RAM, flash memory, ROM (Read OnlyMemory), or any other data storage medium. The memory may include data,programs, and/or instructions and any other software or firmwareexecutable by the processor. Control logic can be implemented inhardware or in software, such as a firmware stored in the memory. Thecontroller 18 controls and monitors the device operation, such asinitialization, configuration, interface, and commands.

The digital camera device or module 10 requires power for its describedfunctions such as for capturing, storing, manipulating, and transmittingthe image. A dedicated power source may be used such as a battery or adedicated connection to an external power source via connector 19. Thepower supply may contain a DC/DC converter. In another embodiment, thepower supply is power fed from the AC power supply via AC plug and acord, and thus may include an AC/DC converter, for converting the ACpower (commonly 115 VAC/60 Hz or 220 VAC/50 Hz) into the required DCvoltage or voltages. Such power supplies are known in the art andtypically involves converting 120 or 240 volt AC supplied by a powerutility company to a well-regulated lower voltage DC for electronicdevices. In one embodiment, the power supply is integrated into a singledevice or circuit, in order to share common circuits. Further, the powersupply may include a boost converter, such as a buck boost converter,charge pump, inverter and regulators as known in the art, as requiredfor conversion of one form of electrical power to another desired formand voltage. While the power supply (either separated or integrated) canbe an integral part and housed within the camera 10 enclosure, it may beenclosed as a separate housing connected via cable to the camera 10assembly. For example, a small outlet plug-in step-down transformershape can be used (also known as wall-wart, “power brick”, “plug pack”,“plug-in adapter”, “adapter block”, “domestic mains adapter”, “poweradapter”, or AC adapter). Further, the power supply may be a linear orswitching type.

Various formats that can be used to represent the captured image areTIFF (Tagged Image File Format), RAW format, AVI, DV, MOV, WMV, MP4, DCF(Design Rule for Camera Format), ITU-T H.261, ITU-T H.263, ITU-T H.264,ITU-T CCM 601, ASF, Exif (Exchangeable Image File Format), and DPOF(Digital Print Order Format) standards. In many cases, video data iscompressed before transmission, in order to allow its transmission overa reduced bandwidth transmission system. The video compressor 14 (orvideo encoder) shown in FIG. 1 is disposed between the image processor13 and the transceiver 15, allowing for compression of the digital videosignal before its transmission over a cable or over-the-air. In somecases, compression may not be required, hence obviating the need forsuch compressor 14. Such compression can be lossy or lossless types.Common compression algorithms are JPEG (Joint Photographic ExpertsGroup) and MPEG (Moving Picture Experts Group). The above and otherimage or video compression techniques can make use of intraframecompression commonly based on registering the differences between partof single frame or a single image. Interframe compression can further beused for video streams, based on registering differences between frames.Other examples of image processing include run length encoding and deltamodulation. Further, the image can be dynamically dithered to allow thedisplayed image to appear to have higher resolution and quality.

The single lens or a lens array 11 is positioned to collect opticalenergy representative of a subject or a scenery, and to focus theoptical energy onto the photosensor array 12. Commonly, the photosensorarray 12 is a matrix of photosensitive pixels, which generates anelectric signal that is a representative of the optical energy directedat the pixel by the imaging optics. The captured image (still images oras video data) may be stored in a memory 17, that may be volatile ornon-volatile memory, and may be a built-in or removable media. Manystand-alone cameras use SD format, while a few use CompactFlash or othertypes. A LCD or TFT miniature display 16 typically serves as anElectronic ViewFinder (EVF) where the image captured by the lens iselectronically displayed. The image on this display is used to assist inaiming the camera at the scene to be photographed. The sensor recordsthe view through the lens; the view is then processed, and finallyprojected on a miniature display, which is viewable through theeyepiece. Electronic viewfinders are used in digital still cameras andin video cameras. Electronic viewfinders can show additionalinformation, such as an image histogram, focal ratio, camera settings,battery charge, and remaining storage space. The display 16 may furtherdisplay images captured earlier that are stored in the memory 17.

A digital camera is described in U.S. Pat. No. 6,897,891 to Itsukaichientitled: “Computer System Using a Camera That is Capable of InputtingMoving Picture or Still Picture Data”, in U.S. Patent ApplicationPublication No. 2007/0195167 to Ishiyama entitled: “Image DistributionSystem, Image Distribution Server, and Image Distribution Method”, inU.S. Patent Application Publication No. 2009/0102940 to Uchida entitled:“Imaging Device and imaging Control Method”, and in U.S. Pat. No.5,798,791 to Katayama et al. entitled: “Multieye Imaging Apparatus”,which are all incorporated in their entirety for all purposes as iffully set forth herein.

A digital camera capable of being set to implement the function of acard reader or camera is disclosed in U.S. Patent ApplicationPublication 2002/0101515 to Yoshida et al. entitled: “Digital camera andMethod of Controlling Operation of Same”, which is incorporated in itsentirety for all purposes as if fully set forth herein. When the digitalcamera capable of being set to implement the function of a card readeror camera is connected to a computer via a USB, the computer is notifiedof the function to which the camera has been set. When the computer andthe digital camera are connected by the USB, a device request istransmitted from the computer to the digital camera. Upon receiving thedevice request, the digital camera determines whether its operation atthe time of the USB connection is that of a card reader or PC camera.Information indicating the result of the determination is incorporatedin a device descriptor, which the digital camera then transmits to thecomputer. Based on the device descriptor, the computer detects the typeof operation to which the digital camera has been set. The driver thatsupports this operation is loaded and the relevant commands aretransmitted from the computer to the digital camera.

A prior art example of a portable electronic camera connectable to acomputer is disclosed in U.S. Pat. No. 5,402,170 to Parulski et al.entitled: “Hand-Manipulated Electronic Camera Tethered to a PersonalComputer”, a digital electronic camera which can accept various types ofinput/output cards or memory cards is disclosed in U.S. Pat. No.7,432,952 to Fukuoka entitled: “Digital Image Capturing Device having anInterface for Receiving a Control Program”, and the use of a disk driveassembly for transferring images out of an electronic camera isdisclosed in U.S. Pat. No. 5,138,459 to Roberts et al., entitled:“Electronic Still Video Camera with Direct Personal Computer (PC)Compatible Digital Format Output”, which are all incorporated in theirentirety for all purposes as if fully set forth herein. A camera withhuman face detection means is disclosed in U.S. Pat. No. 6,940,545 toRay et al., entitled: “Face Detecting Camera and Method”, and in U.S.Patent Application Publication No. 2012/0249768 to Binder entitled:“System and Method for Control Based on Face or Hand Gesture Detection”,which are both incorporated in their entirety for all purposes as iffully set forth herein. A digital still camera is described in anApplication Note No. AN1928/D (Revision 0-20 Feb. 2001) by FreescaleSemiconductor, Inc. entitled: “Roadrunner—Modular digital still camerareference design”, which is incorporated in its entirety for allpurposes as if fully set forth herein.

An imaging method is disclosed in U.S. Pat. No. 8,773,509 to Panentitled: “Imaging Device, Imaging Method and Recording Medium forAdjusting Imaging Conditions of Optical Systems Based on ViewpointImages”, which is incorporated in its entirety for all purposes as iffully set forth herein. The method includes: calculating an amount ofparallax between a reference optical system and an adjustment targetoptical system; setting coordinates of an imaging condition evaluationregion corresponding to the first viewpoint image outputted by thereference optical system; calculating coordinates of an imagingcondition evaluation region corresponding to the second viewpoint imageoutputted by the adjustment target optical system, based on the setcoordinates of the imaging condition evaluation region corresponding tothe first viewpoint image, and on the calculated amount of parallax; andadjusting imaging conditions of the reference optical system and theadjustment target optical system, based on image data in the imagingcondition evaluation region corresponding to the first viewpoint image,at the set coordinates, and on image data in the imaging conditionevaluation region corresponding to the second viewpoint image, at thecalculated coordinates, and outputting the viewpoint images in theadjusted imaging conditions.

A portable hand-holdable digital camera is described in PatentCooperation Treaty (PCT) International Publication Number WO 2012/013914by Adam LOMAS entitled: “Portable Hand-Holdable Digital Camera withRange Finder”, which is incorporated in its entirety for all purposes asif fully set forth herein. The digital camera comprises a camera housinghaving a display, a power button, a shoot button, a flash unit, and abattery compartment; capture means for capturing an image of an objectin two dimensional form and for outputting the captured two-dimensionalimage to the display; first range finder means including a zoomable lensunit supported by the housing for focusing on an object and calculationmeans for calculating a first distance of the object from the lens unitand thus a distance between points on the captured two-dimensional imageviewed and selected on the display; and second range finder meansincluding an emitted-beam range finder on the housing for separatelycalculating a second distance of the object from the emitted-beam rangefinder and for outputting the second distance to the calculation meansof the first range finder means for combination therewith to improvedistance determination accuracy.

A camera having a pointing aid emitter is described in U.S. Pat. No.5,546,156 to McIntyre entitled: “Camera with Pointing Aid”, which isincorporated in its entirety for all purposes as if fully set forthherein. The pointing aid emitter produces a visible beam generallyaligned with the optical axis of the camera objective lens such that thevisible beam illuminates an object in the scene includes a scenemeasurement system that measures an aspect of the scene and an emittercontroller that adjusts the output power of the pointing aid emitter inaccordance with the scene aspect measured by the scene measurementsystem to reduce power consumption and reduce the risk of damage to theobject that is illuminated by the beam. The scene measurement system ofthe camera preferably comprises an ambient light measuring system of acamera automatic exposure system and a distance measuring system of acamera automatic focus system. The emitter preferably comprises a laserlight source that produces a visible laser beam.

A camera that receives light from a field of view, produces signalsrepresentative of the received light, and intermittently reads thesignals to create a photographic image is described in U.S. Pat. No.5,189,463 to Axelrod et al. entitled: “Camera Aiming Mechanism andMethod”, which is incorporated in its entirety for all purposes as iffully set forth herein. The intermittent reading results inintermissions between readings. The invention also includes a radiantenergy source that works with the camera. The radiant energy sourceproduces a beam of radiant energy and projects the beam duringintermissions between readings. The beam produces a light pattern on anobject within or near the camera's field of view, thereby identifying atleast a part of the field of view. The radiant energy source is often alaser and the radiant energy beam is often a laser beam. A detectionmechanism that detects the intermissions and produces a signal thatcauses the radiant energy source to project the radiant energy beam. Thedetection mechanism is typically an electrical circuit including aretriggerable multivibrator or other functionally similar component.

Image. A digital image is a numeric representation (normally binary) ofa two-dimensional image. Depending on whether the image resolution isfixed, it may be of a vector or raster type. Raster images have a finiteset of digital values, called picture elements or pixels. The digitalimage contains a fixed number of rows and columns of pixels, which arethe smallest individual element in an image, holding quantized valuesthat represent the brightness of a given color at any specific point.Typically, the pixels are stored in computer memory as a raster image orraster map, a two-dimensional array of small integers, where thesevalues are commonly transmitted or stored in a compressed form. Theraster images can be created by a variety of input devices andtechniques, such as digital cameras, scanners, coordinate-measuringmachines, seismographic profiling, airborne radar, and more. Commonimage formats include GIF, JPEG, and PNG.

The Graphics Interchange Format (better known by its acronym GIF) is abitmap image format that supports up to 8 bits per pixel for each image,allowing a single image to reference its palette of up to 256 differentcolors chosen from the 24-bit RGB color space. It also supportsanimations and allows a separate palette of up to 256 colors for eachframe. GIF images are compressed using the Lempel-Ziv-Welch (LZW)lossless data compression technique to reduce the file size withoutdegrading the visual quality. The GIF (GRAPHICS INTERCHANGE FORMAT)Standard Version89a is available fromwww.w3.org/Graphics/GIF/spec-gif89a.txt.

JPEG (seen most often with the .jpg or .jpeg filename extension) is acommonly used method of lossy compression for digital images,particularly for those images produced by digital photography. Thedegree of compression can be adjusted, allowing a selectable tradeoffbetween storage size and image quality and typically achieves 10:1compression with little perceptible loss in image quality. JPEG/Exif isthe most common image format used by digital cameras and otherphotographic image capture devices, along with JPEG/JFIF. The term“JPEG” is an acronym for the Joint Photographic Experts Group, whichcreated the standard. JPEG/JFIF supports a maximum image size of65535×65535 pixels—one to four gigapixels (1000 megapixels), dependingon the aspect ratio (from panoramic 3:1 to square). JPEG is standardizedunder as ISO/IEC 10918-1:1994 entitled: “Information technology—Digitalcompression and coding of continuous-tone still images: Requirements andguidelines”.

Portable Network Graphics (PNG) is a raster graphics file format thatsupports lossless data compression that was created as an improvedreplacement for Graphics Interchange Format (GIF), and is the commonlyused lossless image compression format on the Internet. PNG supportspalette-based images (with palettes of 24-bit RGB or 32-bit RGBAcolors), grayscale images (with or without alpha channel), andfull-color non-palette-based RGBimages (with or without alpha channel).PNG was designed for transferring images on the Internet, not forprofessional-quality print graphics, and, therefore, does not supportnon-RGB color spaces such as CMYK. PNG was published as anISO/IEC15948:2004 standard entitled: “Information technology—Computergraphics and image processing—Portable Network Graphics (PNG):Functional specification”.

Further, a digital image acquisition system that includes a portableapparatus for capturing digital images and a digital processingcomponent for detecting, analyzing, invoking subsequent image captures,and informing the photographer regarding motion blur, and reducing thecamera motion blur in an image captured by the apparatus, is describedin U.S. Pat. No. 8,244,053 entitled: “Method and Apparatus forInitiating Subsequent Exposures Based on Determination of MotionBlurring Artifacts”, and in U.S. Pat. No. 8,285,067 entitled: “MethodNotifying Users Regarding Motion Artifacts Based on Image Analysis”,both to Steinberg et al. which are both incorporated in their entiretyfor all purposes as if fully set forth herein.

Furthermore, a camera that has the release button, a timer, a memory anda control part, and the timer measures elapsed time after the depressingof the release button is released, used to prevent a shutter releasemoment to take a good picture from being missed by shortening timerequired for focusing when a release button is depressed again, isdescribed in Japanese Patent Application Publication No. JP2008033200 toHyo Hana entitled: “Camera”, a through image that is read by a facedetection processing circuit, and the face of an object is detected, andis detected again by the face detection processing circuit while halfpressing a shutter button, used to provide an imaging apparatus capableof photographing a quickly moving child without fail, is described in aJapanese Patent Application Publication No. JP2007208922 to UchidaAkihiro entitled: “Imaging Apparatus”, and a digital camera thatexecutes image evaluation processing for automatically evaluating aphotographic image (exposure condition evaluation, contrast evaluation,blur or focus blur evaluation), and used to enable an imagephotographing apparatus such as a digital camera to automaticallycorrect a photographic image, is described in Japanese PatentApplication Publication No. JP2006050494 to Kita Kazunori entitled:“Image Photographing Apparatus”, which are all incorporated in theirentirety for all purposes as if fully set forth herein.

Wireless. Any embodiment herein may be used in conjunction with one ormore types of wireless communication signals and/or systems, forexample, Radio Frequency (RF), Infra-Red (IR), Frequency-DivisionMultiplexing (FDM), Orthogonal FDM (OFDM), Time-Division Multiplexing(TDM), Time-Division Multiple Access (TDMA), Extended TDMA (E-TDMA),General Packet Radio Service (GPRS), extended GPRS, Code-DivisionMultiple Access (CDMA), Wideband CDMA (WCDMA), CDMA 2000, single-carrierCDMA, multi-carrier CDMA, Multi-Carrier Modulation (MDM), DiscreteMulti-Tone (DMT), Bluetooth®, Global Positioning System (GPS), Wi-Fi,Wi-Max, ZigBee™, Ultra-Wideband (UWB), Global System for Mobilecommunication (GSM), 2G, 2.5G, 3G, 3.5G, Enhanced Data rates for GSMEvolution (EDGE), or the like. Any wireless network or wirelessconnection herein may be operating substantially in accordance withexisting IEEE 802.11, 802.11a, 802.11b, 802.11g, 802.11k, 802.11n,802.11r, 802.16, 802.16d, 802.16e, 802.20, 802.21 standards and/orfuture versions and/or derivatives of the above standards. Further, anetwork element (or a device) herein may consist of, be part of, orinclude, a cellular radio-telephone communication system, a cellulartelephone, a wireless telephone, a Personal Communication Systems (PCS)device, a PDA device that incorporates a wireless communication device,or a mobile/portable Global Positioning System (GPS) device. Further, awireless communication may be based on wireless technologies that aredescribed in Chapter 20: “Wireless Technologies” of the publicationnumber 1-587005-001-3 by Cisco Systems, Inc. (July 1999) entitled:“Internetworking Technologies Handbook”, which is incorporated in itsentirety for all purposes as if fully set forth herein. Wirelesstechnologies and networks are further described in a book published 2005by Pearson Education, Inc. William Stallings [ISBN: 0-13-191835-4]entitled: “Wireless Communications and Networks—second Edition”, whichis incorporated in its entirety for all purposes as if fully set forthherein.

Wireless networking typically employs an antenna (a.k.a. aerial), whichis an electrical device that converts electric power into radio waves,and vice versa, connected to a wireless radio transceiver. Intransmission, a radio transmitter supplies an electric currentoscillating at radio frequency to the antenna terminals, and the antennaradiates the energy from the current as electromagnetic waves (radiowaves). In reception, an antenna intercepts some of the power of anelectromagnetic wave in order to produce a low voltage at its terminalsthat is applied to a receiver to be amplified. Typically an antennaconsists of an arrangement of metallic conductors (elements),electrically connected (often through a transmission line) to thereceiver or transmitter. An oscillating current of electrons forcedthrough the antenna by a transmitter will create an oscillating magneticfield around the antenna elements, while the charge of the electronsalso creates an oscillating electric field along the elements. Thesetime-varying fields radiate away from the antenna into space as a movingtransverse electromagnetic field wave. Conversely, during reception, theoscillating electric and magnetic fields of an incoming radio wave exertforce on the electrons in the antenna elements, causing them to moveback and forth, creating oscillating currents in the antenna. Antennascan be designed to transmit and receive radio waves in all horizontaldirections equally (omnidirectional antennas), or preferentially in aparticular direction (directional or high gain antennas). In the lattercase, an antenna may also include additional elements or surfaces withno electrical connection to the transmitter or receiver, such asparasitic elements, parabolic reflectors or horns, which serve to directthe radio waves into a beam or other desired radiation pattern.

ISM. The Industrial, Scientific and Medical (ISM) radio bands are radiobands (portions of the radio spectrum) reserved internationally for theuse of radio frequency (RF) energy for industrial, scientific andmedical purposes other than telecommunications. In general,communications equipment operating in these bands must tolerate anyinterference generated by ISM equipment, and users have no regulatoryprotection from ISM device operation. The ISM bands are defined by theITU-R in 5.138, 5.150, and 5.280 of the Radio Regulations. Individualcountries use of the bands designated in these sections may differ dueto variations in national radio regulations. Because communicationdevices using the ISM bands must tolerate any interference from ISMequipment, unlicensed operations are typically permitted to use thesebands, since unlicensed operation typically needs to be tolerant ofinterference from other devices anyway. The ISM bands share allocationswith unlicensed and licensed operations; however, due to the highlikelihood of harmful interference, licensed use of the bands istypically low. In the United States, uses of the ISM bands are governedby Part 18 of the Federal Communications Commission (FCC) rules, whilePart 15 contains the rules for unlicensed communication devices, eventhose that share ISM frequencies. In Europe, the ETSI is responsible forgoverning ISM bands.

Commonly used ISM bands include a 2.45 GHz band (also known as 2.4 GHzband) that includes the frequency band between 2.400 GHz and 2.500 GHz,a 5.8 GHz band that includes the frequency band 5.725-5.875 GHz, a 24GHz band that includes the frequency band 24.000-24.250 GHz, a 61 GHzband that includes the frequency band 61.000-61.500 GHz, a 122 GHz bandthat includes the frequency band 122.000-123.000 GHz, and a 244 GHz bandthat includes the frequency band 244.000-246.000 GHz.

ZigBee. ZigBee is a standard for a suite of high-level communicationprotocols using small, low-power digital radios based on an IEEE 802standard for Personal Area Network (PAN). Applications include wirelesslight switches, electrical meters with in-home-displays, and otherconsumer and industrial equipment that require a short-range wirelesstransfer of data at relatively low rates. The technology defined by theZigBee specification is intended to be simpler and less expensive thanother WPANs, such as Bluetooth. ZigBee is targeted at Radio-Frequency(RF) applications that require a low data rate, long battery life, andsecure networking. ZigBee has a defined rate of 250 kbps suited forperiodic or intermittent data or a single signal transmission from asensor or input device.

ZigBee builds upon the physical layer and medium access control definedin IEEE standard 802.15.4 (2003 version) for low-rate WPANs. Thespecification further discloses four main components: network layer,application layer, ZigBee Device Objects (ZDOs), andmanufacturer-defined application objects, which allow for customizationand favor total integration. The ZDOs are responsible for a number oftasks, which include keeping of device roles, management of requests tojoin a network, device discovery, and security. Because ZigBee nodes cango from a sleep to active mode in 30 ms or less, the latency can be lowand devices can be responsive, particularly compared to Bluetoothwake-up delays, which are typically around three seconds. ZigBee nodescan sleep most of the time, thus the average power consumption can belower, resulting in longer battery life.

There are three defined types of ZigBee devices: ZigBee Coordinator(ZC), ZigBee Router (ZR), and ZigBee End Device (ZED). ZigBeeCoordinator (ZC) is the most capable device and forms the root of thenetwork tree and might bridge to other networks. There is exactly onedefined ZigBee coordinator in each network, since it is the device thatstarted the network originally. It is able to store information aboutthe network, including acting as the Trust Center & repository forsecurity keys. ZigBee Router (ZR) may be running an application functionas well as may be acting as an intermediate router, passing on data fromother devices. ZigBee End Device (ZED) contains functionality to talk toa parent node (either the coordinator or a router). This relationshipallows the node to be asleep a significant amount of the time, therebygiving long battery life. A ZED requires the least amount of memory, andtherefore can be less expensive to manufacture than a ZR or ZC.

The protocols build on recent algorithmic research (Ad-hoc On-demandDistance Vector, neuRFon) to automatically construct a low-speed ad-hocnetwork of nodes. In most large network instances, the network will be acluster of clusters. It can also form a mesh or a single cluster. Thecurrent ZigBee protocols support beacon and non-beacon enabled networks.In non-beacon-enabled networks, an unslotted CSMA/CA channel accessmechanism is used. In this type of network, ZigBee Routers typicallyhave their receivers continuously active, requiring a more robust powersupply. However, this allows for heterogeneous networks in which somedevices receive continuously, while others only transmit when anexternal stimulus is detected.

In beacon-enabled networks, the special network nodes called ZigBeeRouters transmit periodic beacons to confirm their presence to othernetwork nodes. Nodes may sleep between the beacons, thus lowering theirduty cycle and extending their battery life. Beacon intervals depend onthe data rate; they may range from 15.36 milliseconds to 251.65824seconds at 250 Kbit/s, from 24 milliseconds to 393.216 seconds at 40Kbit/s, and from 48 milliseconds to 786.432 seconds at 20 Kbit/s. Ingeneral, the ZigBee protocols minimize the time the radio is on toreduce power consumption. In beaconing networks, nodes only need to beactive while a beacon is being transmitted. In non-beacon-enablednetworks, power consumption is decidedly asymmetrical: some devices arealways active while others spend most of their time sleeping.

Except for the Smart Energy Profile 2.0, current ZigBee devices conformto the IEEE 802.15.4-2003 Low-Rate Wireless Personal Area Network(LR-WPAN) standard. The standard specifies the lower protocol layers—thePHYsical layer (PHY), and the Media Access Control (MAC) portion of theData Link Layer (DLL). The basic channel access mode is “Carrier Sense,Multiple Access/Collision Avoidance” (CSMA/CA), that is, the nodes talkin the same way that people converse; they briefly check to see that noone is talking before they start. There are three notable exceptions tothe use of CSMA. Beacons are sent on a fixed time schedule, and do notuse CSMA. Message acknowledgments also do not use CSMA. Finally, devicesin Beacon Oriented networks that have low latency real-time requirement,may also use Guaranteed Time Slots (GTS), which by definition do not useCSMA.

Z-Wave. Z-Wave is a wireless communications protocol by the Z-WaveAlliance (http://www.z-wave.com) designed for home automation,specifically for remote control applications in residential and lightcommercial environments. The technology uses a low-power RF radioembedded or retrofitted into home electronics devices and systems, suchas lighting, home access control, entertainment systems and householdappliances. Z-Wave communicates using a low-power wireless technologydesigned specifically for remote control applications. Z-Wave operatesin the sub-gigahertz frequency range, around 900 MHz. This band competeswith some cordless telephones and other consumer electronics devices,but avoids interference with WiFi and other systems that operate on thecrowded 2.4 GHz band. Z-Wave is designed to be easily embedded inconsumer electronics products, including battery-operated devices suchas remote controls, smoke alarms, and security sensors.

Z-Wave is a mesh networking technology where each node or device on thenetwork is capable of sending and receiving control commands throughwalls or floors, and use intermediate nodes to route around householdobstacles or radio dead spots that might occur in the home. Z-Wavedevices can work individually or in groups, and can be programmed intoscenes or events that trigger multiple devices, either automatically orvia remote control. The Z-wave radio specifications include bandwidth of9,600 bit/s or 40 Kbit/s, fully interoperable, GFSK modulation, and arange of approximately 100 feet (or 30 meters) assuming “open air”conditions, with reduced range indoors depending on building materials,etc. The Z-Wave radio uses the 900 MHz ISM band: 908.42 MHz (UnitedStates); 868.42 MHz (Europe); 919.82 MHz (Hong Kong); and 921.42 MHz(Australia/New Zealand).

Z-Wave uses a source-routed mesh network topology and has one or moremaster controllers that control routing and security. The devices cancommunicate to another by using intermediate nodes to actively routearound, and circumvent household obstacles or radio dead spots thatmight occur. A message from node A to node C can be successfullydelivered even if the two nodes are not within range, providing that athird node B can communicate with nodes A and C. If the preferred routeis unavailable, the message originator will attempt other routes until apath is found to the “C” node. Therefore, a Z-Wave network can span muchfarther than the radio range of a single unit; however, with several ofthese hops, a delay may be introduced between the control command andthe desired result. In order for Z-Wave units to be able to routeunsolicited messages, they cannot be in sleep mode. Therefore, mostbattery-operated devices are not designed as repeater units. A Z-Wavenetwork can consist of up to 232 devices with the option of bridgingnetworks if more devices are required.

WWAN. Any wireless network herein may be a Wireless Wide Area Network(WWAN) such as a wireless broadband network, and the WWAN port may be anantenna and the WWAN transceiver may be a wireless modem. The wirelessnetwork may be a satellite network, the antenna may be a satelliteantenna, and the wireless modem may be a satellite modem. The wirelessnetwork may be a WiMAX network such as according to, compatible with, orbased on, IEEE 802.16-2009, the antenna may be a WiMAX antenna, and thewireless modem may be a WiMAX modem. The wireless network may be acellular telephone network, the antenna may be a cellular antenna, andthe wireless modem may be a cellular modem. The cellular telephonenetwork may be a Third Generation (3G) network, and may use UMTS W-CDMA,UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, or GSMEDGE-Evolution. The cellular telephone network may be a FourthGeneration (4G) network and may use or be compatible with HSPA+, MobileWiMAX, LTE, LTE-Advanced, MBWA, or may be compatible with, or based on,IEEE 802.20-2008.

WLAN. Wireless Local Area Network (WLAN), is a popular wirelesstechnology that makes use of the Industrial, Scientific and Medical(ISM) frequency spectrum. In the US, three of the bands within the ISMspectrum are the A band, 902-928 MHz; the B band, 2.4-2.484 GHz (a.k.a.2.4 GHz); and the C band, 5.725-5.875 GHz (a.k.a. 5 GHz). Overlappingand/or similar bands are used in different regions such as Europe andJapan. In order to allow interoperability between equipment manufacturedby different vendors, few WLAN standards have evolved, as part of theIEEE 802.11 standard group, branded as WiFi (www.wi-fi.org). IEEE802.11b describes a communication using the 2.4 GHz frequency band andsupporting communication rate of 11 Mb/s, IEEE 802.11a uses the 5 GHzfrequency band to carry 54 MB/s and IEEE 802.11g uses the 2.4 GHz bandto support 54 Mb/s. The WiFi technology is further described in apublication entitled: “WiFi Technology” by Telecom Regulatory Authority,published on July 2003, which is incorporated in its entirety for allpurposes as if fully set forth herein. The IEEE 802 defines an ad-hocconnection between two or more devices without using a wireless accesspoint: the devices communicate directly when in range. An ad hoc networkoffers peer-to-peer layout and is commonly used in situations such as aquick data exchange or a multiplayer LAN game, because the setup is easyand an access point is not required.

A node/client with a WLAN interface is commonly referred to as STA(Wireless Station/Wireless client). The STA functionality may beembedded as part of the data unit, or alternatively be a dedicated unit,referred to as bridge, coupled to the data unit. While STAs maycommunicate without any additional hardware (ad-hoc mode), such networkusually involves Wireless Access Point (a.k.a. WAP or AP) as a mediationdevice. The WAP implements the Basic Stations Set (BSS) and/or ad-hocmode based on Independent BSS (IBSS). STA, client, bridge and WAP willbe collectively referred to hereon as WLAN unit. Bandwidth allocationfor IEEE 802.11g wireless in the U.S. allows multiple communicationsessions to take place simultaneously, where eleven overlapping channelsare defined spaced 5 MHz apart, spanning from 2412 MHz as the centerfrequency for channel number 1, via channel 2 centered at 2417 MHz and2457 MHz as the center frequency for channel number 10, up to channel 11centered at 2462 MHz. Each channel bandwidth is 22 MHz, symmetrically(+/−11 MHz) located around the center frequency. In the transmissionpath, first the baseband signal (IF) is generated based on the data tobe transmitted, using 256 QAM (Quadrature Amplitude Modulation) basedOFDM (Orthogonal Frequency Division Multiplexing) modulation technique,resulting a 22 MHz (single channel wide) frequency band signal. Thesignal is then up converted to the 2.4 GHz (RF) and placed in the centerfrequency of required channel, and transmitted to the air via theantenna. Similarly, the receiving path comprises a received channel inthe RF spectrum, down converted to the baseband (IF) wherein the data isthen extracted.

In order to support multiple devices and using a permanent solution, aWireless Access Point (WAP) is typically used. A Wireless Access Point(WAP, or Access Point—AP) is a device that allows wireless devices toconnect to a wired network using Wi-Fi, or related standards. The WAPusually connects to a router (via a wired network) as a standalonedevice, but can also be an integral component of the router itself.Using Wireless Access Point (AP) allows users to add devices that accessthe network with little or no cables. A WAP normally connects directlyto a wired Ethernet connection, and the AP then provides wirelessconnections using radio frequency links for other devices to utilizethat wired connection. Most APs support the connection of multiplewireless devices to one wired connection. Wireless access typicallyinvolves special security considerations, since any device within arange of the WAP can attach to the network. The most common solution iswireless traffic encryption. Modern access points come with built-inencryption such as Wired Equivalent Privacy (WEP) and Wi-Fi ProtectedAccess (WPA), typically used with a password or a passphrase.Authentication in general, and a WAP authentication in particular, isused as the basis for authorization, which determines whether aprivilege may be granted to a particular user or process, privacy, whichkeeps information from becoming known to non-participants, andnon-repudiation, which is the inability to deny having done somethingthat was authorized to be done based on the authentication. Anauthentication in general, and a WAP authentication in particular, mayuse an authentication server that provides a network service thatapplications may use to authenticate the credentials, usually accountnames and passwords of their users. When a client submits a valid set ofcredentials, it receives a cryptographic ticket that it can subsequentlybe used to access various services. Authentication algorithms includepasswords, Kerberos, and public key encryption.

Prior art technologies for data networking may be based on singlecarrier modulation techniques, such as AM (Amplitude Modulation), FM(Frequency Modulation), and PM (Phase Modulation), as well as bitencoding techniques such as QAM (Quadrature Amplitude Modulation) andQPSK (Quadrature Phase Shift Keying). Spread spectrum technologies, toinclude both DSSS (Direct Sequence Spread Spectrum) and FHSS (FrequencyHopping Spread Spectrum) are known in the art. Spread spectrum commonlyemploys Multi-Carrier Modulation (MCM) such as OFDM (OrthogonalFrequency Division Multiplexing). OFDM and other spread spectrum arecommonly used in wireless communication systems, particularly in WLANnetworks.

Bluetooth. Bluetooth is a wireless technology standard for exchangingdata over short distances (using short-wavelength UHF radio waves in theISM band from 2.4 to 2.485 GHz) from fixed and mobile devices, andbuilding personal area networks (PANs). It can connect several devices,overcoming problems of synchronization. A Personal Area Network (PAN)may be according to, compatible with, or based on, Bluetooth™ or IEEE802.15.1-2005 standard. A Bluetooth controlled electrical appliance isdescribed in U.S. Patent Application No. 2014/0159877 to Huang entitled:“Bluetooth Controllable Electrical Appliance”, and an electric powersupply is described in U.S. Patent Application No. 2014/0070613 to Garbet al. entitled: “Electric Power Supply and Related Methods”, which areboth incorporated in their entirety for all purposes as if fully setforth herein. Any Personal Area Network (PAN) may be according to,compatible with, or based on, Bluetooth™ or IEEE 802.15.1-2005 standard.A Bluetooth controlled electrical appliance is described in U.S. PatentApplication No. 2014/0159877 to Huang entitled: “Bluetooth ControllableElectrical Appliance”, and an electric power supply is described in U.S.Patent Application No. 2014/0070613 to Garb et al. entitled: “ElectricPower Supply and Related Methods”, which are both incorporated in theirentirety for all purposes as if fully set forth herein.

Bluetooth operates at frequencies between 2402 and 2480 MHz, or 2400 and2483.5 MHz including guard bands 2 MHz wide at the bottom end and 3.5MHz wide at the top. This is in the globally unlicensed (but notunregulated) Industrial, Scientific and Medical (ISM) 2.4 GHzshort-range radio frequency band. Bluetooth uses a radio technologycalled frequency-hopping spread spectrum. Bluetooth divides transmitteddata into packets, and transmits each packet on one of 79 designatedBluetooth channels. Each channel has a bandwidth of 1 MHz. It usuallyperforms 800 hops per second, with Adaptive Frequency-Hopping (AFH)enabled. Bluetooth low energy uses 2 MHz spacing, which accommodates 40channels. Bluetooth is a packet-based protocol with a master-slavestructure. One master may communicate with up to seven slaves in apiconet. All devices share the master's clock. Packet exchange is basedon the basic clock, defined by the master, which ticks at 312.5 μsintervals. Two clock ticks make up a slot of 625 μs, and two slots makeup a slot pair of 1250 μs. In the simple case of single-slot packets themaster transmits in even slots and receives in odd slots. The slave,conversely, receives in even slots and transmits in odd slots. Packetsmay be 1, 3 or 5 slots long, but in all cases the master's transmissionbegins in even slots and the slave's in odd slots.

A master Bluetooth device can communicate with a maximum of sevendevices in a piconet (an ad-hoc computer network using Bluetoothtechnology), though not all devices reach this maximum. The devices canswitch roles, by agreement, and the slave can become the master (forexample, a headset initiating a connection to a phone necessarily beginsas master—as initiator of the connection—but may subsequently operate asslave). The Bluetooth Core Specification provides for the connection oftwo or more piconets to form a scatternet, in which certain devicessimultaneously play the master role in one piconet and the slave role inanother. At any given time, data can be transferred between the masterand one other device (except for the little-used broadcast mode). Themaster chooses which slave device to address; typically, it switchesrapidly from one device to another in a round-robin fashion. Since it isthe master that chooses which slave to address, whereas a slave issupposed to listen in each receive slot, being a master is a lighterburden than being a slave. Being a master of seven slaves is possible;being a slave of more than one master is difficult.

Bluetooth Low Energy. Bluetooth low energy (Bluetooth LE, BLE, marketedas Bluetooth Smart) is a wireless personal area network technologydesigned and marketed by the Bluetooth Special Interest Group (SIG)aimed at novel applications in the healthcare, fitness, beacons,security, and home entertainment industries. Compared to ClassicBluetooth, Bluetooth Smart is intended to provide considerably reducedpower consumption and cost while maintaining a similar communicationrange. Bluetooth low energy is described in a Bluetooth SIG publishedDec. 2, 2014 standard Covered Core Package version: 4.2, entitled:“Master Table of Contents & Compliance Requirements—Specification Volume0”, and in an article published 2012 in Sensors [ISSN 1424-8220] byCaries Gomez et al. [Sensors 2012, 12, 11734-11753;doi:10.3390/s120211734] entitled: “Overview and Evaluation of BluetoothLow Energy: An Emerging Low-Power Wireless Technology”, which are bothincorporated in their entirety for all purposes as if fully set forthherein.

Bluetooth Smart technology operates in the same spectrum range (the2.400 GHz-2.4835 GHz ISM band) as Classic Bluetooth technology, but usesa different set of channels. Instead of the Classic Bluetooth 79 1-MHzchannels, Bluetooth Smart has 40 2-MHz channels. Within a channel, datais transmitted using Gaussian frequency shift modulation, similar toClassic Bluetooth's Basic Rate scheme. The bit rate is 1 Mbit/s, and themaximum transmit power is 10 mW. Bluetooth Smart uses frequency hoppingto counteract narrowband interference problems. Classic Bluetooth alsouses frequency hopping but the details are different; as a result, whileboth FCC and ETSI classify Bluetooth technology as an FHSS scheme,Bluetooth Smart is classified as a system using digital modulationtechniques or a direct-sequence spread spectrum. All Bluetooth Smartdevices use the Generic Attribute Profile (GATT). The applicationprogramming interface offered by a Bluetooth Smart aware operatingsystem will typically be based around GATT concepts.

Cellular. Cellular telephone network may be according to, compatiblewith, or may be based on, a Third Generation (3G) network that uses UMTSW-CDMA, UMTS HSPA, UMTS TDD, CDMA2000 1×RTT, CDMA2000 EV-DO, or GSMEDGE-Evolution. The cellular telephone network may be a FourthGeneration (4G) network that uses HSPA+, Mobile WiMAX, LTE,LTE-Advanced, MBWA, or may be based on or compatible with IEEE802.20-2008.

DSRC. Dedicated Short-Range Communication (DSRC) is a one-way or two-wayshort-range to medium-range wireless communication channels specificallydesigned for automotive use and a corresponding set of protocols andstandards. DSRC is a two-way short-to-medium range wirelesscommunications capability that permits very high data transmissioncritical in communications-based active safety applications. In Reportand Order FCC-03-324, the Federal Communications Commission (FCC)allocated 75 MHz of spectrum in the 5.9 GHz band for use by intelligenttransportations systems (ITS) vehicle safety and mobility applications.DSRC serves a short to medium range (1000 meters) communications serviceand supports both public safety and private operations inroadside-to-vehicle and vehicle-to-vehicle communication environments byproviding very high data transfer rates where minimizing latency in thecommunication link and isolating relatively small communication zones isimportant. DSRC transportation applications for Public Safety andTraffic Management include Blind spot warnings, Forward collisionwarnings, Sudden braking ahead warnings, Do not pass warnings,Intersection collision avoidance and movement assistance, Approachingemergency vehicle warning, Vehicle safety inspection, Transit oremergency vehicle signal priority, Electronic parking and toll payments,Commercial vehicle clearance and safety inspections, In-vehicle signing,Rollover warning, and Traffic and travel condition data to improvetraveler information and maintenance services.

The European standardization organization European Committee forStandardization (CEN), sometimes in co-operation with the InternationalOrganization for Standardization (ISO) developed some DSRC standards: EN12253:2004 Dedicated Short-Range Communication—Physical layer usingmicrowave at 5.8 GHz (review), EN 12795:2002 Dedicated Short-RangeCommunication (DSRC)—DSRC Data link layer: Medium Access and LogicalLink Control (review), EN 12834:2002 Dedicated Short-RangeCommunication—Application layer (review), EN 13372:2004 DedicatedShort-Range Communication (DSRC)—DSRC profiles for RTTT applications(review), and EN ISO 14906:2004 Electronic Fee Collection—Applicationinterface. An overview of the DSRC/WAVE technologies is described in apaper by Yunxin (Jeff) Li (Eveleigh, NSW 2015, Australia) downloadedfrom the Internet on July 2017, entitled: “An Overview of the DSRC/WAVETechnology”, and the DSRC is further standardized as ARIB STD—T75VERSION 1.0, published September 2001 by Association of Radio Industriesand Businesses Kasumigaseki, Chiyoda-ku, Tokyo 100-0013, Japan,entitled: “DEDICATED SHORT-RANGE COMMUNICATION SYSTEM—ARIB STANDARDVersion 1.0”, which are both incorporated in their entirety for allpurposes as if fully set forth herein.

IEEE 802.11p. The IEEE 802.11p standard is an example of DSRC and is apublished standard entitled: “Part 11: Wireless LAN Medium AccessControl (MAC) and Physical Layer (PHY) Specifications Amendment 6:Wireless Access in Vehicular Environments”, that adds wireless access invehicular environments (WAVE), a vehicular communication system, forsupporting Intelligent Transportation Systems (ITS) applications. Itincludes data exchange between high-speed vehicles and between thevehicles and the roadside infrastructure, so called V2X communication,in the licensed ITS band of 5.9 GHz (5.85-5.925 GHz). IEEE 1609 is ahigher layer standard based on the IEEE 802.11p, and is also the base ofa European standard for vehicular communication known as ETSI ITS-G5.2.The Wireless Access in Vehicular Environments (WAVE/DSRC) architectureand services necessary for multi-channel DSRC/WAVE devices tocommunicate in a mobile vehicular environment is described in the familyof IEEE 1609 standards, such as IEEE 1609.1-2006 Resource Manager, IEEEStd 1609.2 Security Services for Applications and Management Messages,IEEE Std 1609.3 Networking Services, IEEE Std 1609.4 Multi-ChannelOperation IEEE Std 1609.5 Communications Manager, as well as IEEEP802.11p Amendment: “Wireless Access in Vehicular Environments”.

As the communication link between the vehicles and the roadsideinfrastructure might exist for only a short amount of time, the IEEE802.11p amendment defines a way to exchange data through that linkwithout the need to establish a Basic Service Set (BSS), and thus,without the need to wait for the association and authenticationprocedures to complete before exchanging data. For that purpose, IEEE802.11p enabled stations use the wildcard BSSID (a value of all 1s) inthe header of the frames they exchange, and may start sending andreceiving data frames as soon as they arrive on the communicationchannel. Because such stations are neither associated nor authenticated,the authentication and data confidentiality mechanisms provided by theIEEE 802.11 standard (and its amendments) cannot be used. These kinds offunctionality must then be provided by higher network layers. IEEE802.11p standard uses channels within the 75 MHz bandwidth in the 5.9GHz band (5.850-5.925 GHz). This is half the bandwidth, or double thetransmission time for a specific data symbol, as used in 802.11a. Thisallows the receiver to better cope with the characteristics of the radiochannel in vehicular communications environments, e.g., the signalechoes reflected from other cars or houses.

Compression. Data compression, also known as source coding and bit-ratereduction, involves encoding information using fewer bits than theoriginal representation. Compression can be either lossy, or lossless.Lossless compression reduces bits by identifying and eliminatingstatistical redundancy, so that no information is lost in losslesscompression. Lossy compression reduces bits by identifying unnecessaryinformation and removing it. The process of reducing the size of a datafile is commonly referred to as a data compression. A compression isused to reduce resource usage, such as data storage space, ortransmission capacity. Data compression is further described in aCarnegie Mellon University chapter entitled: “Introduction to DataCompression” by Guy E. Blelloch, dated Jan. 31, 2013, which isincorporated in its entirety for all purposes as if fully set forthherein.

In a scheme involving lossy data compression, some loss of informationis acceptable. For example, dropping of a nonessential detail from adata can save storage space. Lossy data compression schemes may beinformed by research on how people perceive the data involved. Forexample, the human eye is more sensitive to subtle variations inluminance than it is to variations in color. JPEG image compressionworks in part by rounding off nonessential bits of information. There isa corresponding trade-off between preserving information and reducingsize. A number of popular compression formats exploit these perceptualdifferences, including those used in music files, images, and video.

Lossy image compression is commonly used in digital cameras, to increasestorage capacities with minimal degradation of picture quality.Similarly, DVDs use the lossy MPEG-2 Video codec for video compression.In lossy audio compression, methods of psychoacoustics are used toremove non-audible (or less audible) components of the audio signal.Compression of human speech is often performed with even morespecialized techniques, speech coding, or voice coding, is sometimesdistinguished as a separate discipline from audio compression. Differentaudio and speech compression standards are listed under audio codecs.Voice compression is used in Internet telephony, for example, and audiocompression is used for CD ripping and is decoded by audio player.

Lossless data compression algorithms usually exploit statisticalredundancy to represent data more concisely without losing information,so that the process is reversible. Lossless compression is possiblebecause most real-world data has statistical redundancy. The Lempel-Ziv(LZ) compression methods are among the most popular algorithms forlossless storage. DEFLATE is a variation on LZ optimized fordecompression speed and compression ratio, and is used in PKZIP, Gzipand PNG. The LZW (Lempel-Ziv-Welch) method is commonly used in GIFimages, and is described in IETF RFC 1951. The LZ methods use atable-based compression model where table entries are substituted forrepeated strings of data. For most LZ methods, this table is generateddynamically from earlier data in the input. The table itself is oftenHuffman encoded (e.g., SHRI, LZX). Typical modern lossless compressorsuse probabilistic models, such as prediction by partial matching.

Lempel-Ziv-Welch (LZW) is an example of lossless data compressionalgorithm created by Abraham Lempel, Jacob Ziv, and Terry Welch. Thealgorithm is simple to implement, and has the potential for very highthroughput in hardware implementations. It was the algorithm of thewidely used Unix file compression utility compress, and is used in theGIF image format. The LZW and similar algorithms are described in U.S.Pat. No. 4,464,650 to Eastman et al. entitled: “Apparatus and Method forCompressing Data Signals and Restoring the Compressed Data Signals”, inU.S. Pat. No. 4,814,746 to Miller et al. entitled: “Data CompressionMethod”, and in U.S. Pat. No. 4,558,302 to Welch entitled: “High SpeedData Compression and Decompression Apparatus and Method”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

Image/video. Any content herein may consist of, be part of, or include,an image or a video content. A video content may be in a digital videoformat that may be based on one out of: TIFF (Tagged Image File Format),RAW format, AVI, DV, MOV, WMV, MP4, DCF (Design Rule for Camera Format),ITU-T H.261, ITU-T H.263, ITU-T H.264, ITU-T CCIR 601, ASF, Exif(Exchangeable Image File Format), and DPOF (Digital Print Order Format)standards. An intraframe or interframe compression may be used, and thecompression may be a lossy or a non-lossy (lossless) compression, thatmay be based on a standard compression algorithm, which may be one ormore out of JPEG (Joint Photographic Experts Group) and MPEG (MovingPicture Experts Group), ITU-T H.261, ITU-T H.263, ITU-T H.264 and ITU-TCCIR 601.

Video. The term ‘video’ typically pertains to numerical or electricalrepresentation or moving visual images, commonly referring to recording,reproducing, displaying, or broadcasting the moving visual images.Video, or a moving image in general, is created from a sequence of stillimages called frames, and by recording and then playing back frames inquick succession, an illusion of movement is created. Video can beedited by removing some frames and combining sequences of frames, calledclips, together in a timeline. A Codec, short for ‘coder-decoder’,describes the method in which video data is encoded into a file anddecoded when the file is played back. Most video is compressed duringencoding, and so the terms codec and compressor are often usedinterchangeably. Codecs can be lossless or lossy, where lossless codecsare higher quality than lossy codecs, but produce larger file sizes.Transcoding is the process of converting from one codec to another.Common codecs include DV-PAL, HDV, H.264, MPEG-2, and MPEG-4. Digitalvideo is further described in Adobe Digital Video Group publicationupdated and enhanced March 2004, entitled: “A Digital Video Primer—Anintroduction to DV production, post-production, and delivery”, which isincorporated in its entirety for all purposes as if fully set forthherein.

Digital video data typically comprises a series of frames, includingorthogonal bitmap digital images displayed in rapid succession at aconstant rate, measured in Frames-Per-Second (FPS). In interlaced videoeach frame is composed of two halves of an image (referred toindividually as fields, two consecutive fields compose a full frame),where the first half contains only the odd-numbered lines of a fullframe, and the second half contains only the even-numbered lines.

Many types of video compression exist for serving digital video over theinternet, and on optical disks. The file sizes of digital video used forprofessional editing are generally not practical for these purposes, andthe video requires further compression with codecs such as Sorenson,H.264, and more recently, Apple ProRes especially for HD. Currentlywidely used formats for delivering video over the internet are MPEG-4,Quicktime, Flash, and Windows Media. Other PCM based formats include CCM601 commonly used for broadcast stations, MPEG-4 popular for onlinedistribution of large videos and video recorded to flash memory, MPEG-2used for DVDs, Super-VCDs, and many broadcast television formats, MPEG-1typically used for video CDs, and H.264 (also known as MPEG-4 Part 10 orAVC) commonly used for Blu-ray Discs and some broadcast televisionformats.

The term ‘Standard Definition’ (SD) describes the frame size of a video,typically having either a 4:3 or 16:9 frame aspect ratio. The SD PALstandard defines 4:3 frame size and 720×576 pixels, (or 768×576 if usingsquare pixels), while SD web video commonly uses a frame size of 640×480pixels. Standard-Definition Television (SDTV) refers to a televisionsystem that uses a resolution that is not considered to be eitherhigh-definition television (1080i, 1080p, 1440p, 4K UHDTV, and 8K UHD)or enhanced-definition television (EDTV 480p). The two common SDTVsignal types are 576i, with 576 interlaced lines of resolution, derivedfrom the European-developed PAL and SECAM systems, and 480i based on theAmerican National Television System Committee NTSC system. In NorthAmerica, digital SDTV is broadcast in the same 4:3 aspect ratio as NTSCsignals with widescreen content being center cut. However, in otherparts of the world that used the PAL or SECAM color systems,standard-definition television is now usually shown with a 16:9 aspectratio. Standards that support digital SDTV broadcast include DVB, ATSC,and ISDB.

The term ‘High-Definition’ (HD) refers multiple video formats, which usedifferent frame sizes, frame rates and scanning methods, offering higherresolution and quality than standard-definition. Generally, any videoimage with considerably more than 480 horizontal lines (North America)or 576 horizontal lines (Europe) is considered high-definition, where720 scan lines is commonly the minimum. HD video uses a 16:9 frameaspect ratio and frame sizes that are 1280×720 pixels (used for HDtelevision and HD web video), 1920×1080 pixels (referred to as full-HDor full-raster), or 1440×1080 pixels (full-HD with non-square pixels).

High definition video (prerecorded and broadcast) is defined by thenumber of lines in the vertical display resolution, such as 1,080 or 720lines, in contrast to regular digital television (DTV) using 480 lines(upon which NTSC is based, 480 visible scanlines out of 525) or 576lines (upon which PAL/SECAM are based, 576 visible scanlines out of625). HD is further defined by the scanning system being progressivescanning (p) or interlaced scanning (i). Progressive scanning (p)redraws an image frame (all of its lines) when refreshing each image,for example 720p/1080p. Interlaced scanning (i) draws the image fieldevery other line or “odd numbered” lines during the first image refreshoperation, and then draws the remaining “even numbered” lines during asecond refreshing, for example 1080i. Interlaced scanning yields greaterimage resolution if a subject is not moving, but loses up to half of theresolution, and suffers “combing” artifacts when a subject is moving. HDvideo is further defined by the number of frames (or fields) per second(Hz), where in Europe 50 Hz (60 Hz in the USA) television broadcastingsystem is common. The 720p60 format is 1,280×720 pixels, progressiveencoding with 60 frames per second (60 Hz). The 1080i50/1080i60 formatis 1920×1080 pixels, interlaced encoding with 50/60 fields, (50/60 Hz)per second.

Currently common HD modes are defined as 720p, 1080i, 1080p, and 1440p.Video mode 720p relates to frame size of 1,280×720 (W×H) pixels, 921,600pixels per image, progressive scanning, and frame rates of 23.976, 24,25, 29.97, 30, 50, 59.94, 60, or 72 Hz. Video mode 1080i relates toframe size of 1,920×1,080 (W×H) pixels, 2,073,600 pixels per image,interlaced scanning, and frame rates of 25 (50 fields/s), 29.97 (59.94fields/s), or 30 (60 fields/s) Hz. Video mode 1080p relates to framesize of 1,920×1,080 (W×H) pixels, 2,073,600 pixels per image,progressive scanning, and frame rates of 24 (23.976), 25, 30 (29.97),50, or 60 (59.94) Hz. Similarly, video mode 1440p relates to frame sizeof 2,560×1,440 (W×H) pixels, 3,686,400 pixels per image, progressivescanning, and frame rates of 24 (23.976), 25, 30 (29.97), 50, or 60(59.94) Hz. Digital video standards are further described in a published2009 primer by Tektronix® entitled: “A Guide to Standard andHigh-Definition Digital Video Measurements”, which is incorporated inits entirety for all purposes as if fully set forth herein.

MPEG-4. MPEG-4 is a method of defining compression of audio and visual(AV) digital data, designated as a standard for a group of audio andvideo coding formats, and related technology by the IS O/IEC MovingPicture Experts Group (MPEG) (ISO/IEC JTC1/SC29/WG11) under the formalstandard ISO/IEC 14496—‘Coding of audio-visual objects’. Typical uses ofMPEG-4 include compression of AV data for the web (streaming media) andCD distribution, voice (telephone, videophone) and broadcast televisionapplications. MPEG-4 provides a series of technologies for developers,for various service-providers and for end users, as well as enablingdevelopers to create multimedia objects possessing better abilities ofadaptability and flexibility to improve the quality of such services andtechnologies as digital television, animation graphics, the World WideWeb and their extensions. Transporting of MPEG-4 is described in IETFRFC 3640, entitled: “RTP Payload Format for Transport of MPEG-4Elementary Streams”, which is incorporated in its entirety for allpurposes as if fully set forth herein. The MPEG-4 format can performvarious functions such as multiplexing and synchronizing data,associating with media objects for efficiently transporting via variousnetwork channels. MPEG-4 is further described in a white paper published2005 by The MPEG Industry Forum (Document Number mp-in-40182), entitled:“Understanding MPEG-4: Technologies, Advantages, and Markets—An MPEGIFWhite Paper”, which is incorporated in its entirety for all purposes asif fully set forth herein.

H.264. H.264 (a.k.a. MPEG-4 Part 10, or Advanced Video Coding (MPEG-4AVC)) is a commonly used video compression format for the recording,compression, and distribution of video content. H.264/MPEG-4 AVC is ablock-oriented motion-compensation-based video compression standardITU-T H.264, developed by the ITU-T Video Coding Experts Group (VCEG)together with the ISO/IEC JTC1 Moving Picture Experts Group (MPEG),defined in the ISO/IEC MPEG-4 AVC standard ISO/IEC 14496-10—MPEG-4 Part10—‘Advanced Video Coding’. H.264 is widely used by streaming internetsources, such as videos from Vimeo, YouTube, and the iTunes Store, websoftware such as the Adobe Flash Player and Microsoft Silverlight, andalso various HDTV broadcasts over terrestrial (ATSC, ISDB-T, DVB-T orDVB-T2), cable (DVB-C), and satellite (DVB-S and DVB-S2). H.264 isfurther described in a Standards Report published in IEEE CommunicationsMagazine, August 2006, by Gary J. Sullivan of Microsoft Corporation,entitled: “The H.264/MPEG4 Advanced Video Coding Standard and itsApplications”, and further in IETF RFC 3984 entitled: “RTP PayloadFormat for H.264 Video”, which are both incorporated in their entiretyfor all purposes as if fully set forth herein.

VCA. Video Content Analysis (VCA), also known as video contentanalytics, is the capability of automatically analyzing video to detectand determine temporal and spatial events. VCA deals with the extractionof metadata from raw video to be used as components for furtherprocessing in applications such as search, summarization, classificationor event detection. The purpose of video content analysis is to provideextracted features and identification of structure that constitutebuilding blocks for video retrieval, video similarity finding,summarization and navigation. Video content analysis transforms theaudio and image stream into a set of semantically meaningfulrepresentations. The ultimate goal is to extract structural and semanticcontent automatically, without any human intervention, at least forlimited types of video domains. Algorithms to perform content analysisinclude those for detecting objects in video, recognizing specificobjects, persons, locations, detecting dynamic events in video,associating keywords with image regions or motion. VCA is used in a widerange of domains including entertainment, healthcare, retail,automotive, transport, home automation, flame and smoke detection,safety and security. The algorithms can be implemented as software ongeneral purpose machines, or as hardware in specialized video processingunits.

Many different functionalities can be implemented in VCA. Video MotionDetection is one of the simpler forms where motion is detected withregard to a fixed background scene. More advanced functionalitiesinclude video tracking and egomotion estimation. Based on the internalrepresentation that VCA generates in the machine, it is possible tobuild other functionalities, such as identification, behavior analysisor other forms of situation awareness. VCA typically relies on goodinput video, so it is commonly combined with video enhancementtechnologies such as video denoising, image stabilization, unsharpmasking and super-resolution. VCA is described in a publicationentitled: “An introduction to video content analysis—industry guide”published August 2016 as Form No. 262 Issue 2 by British SecurityIndustry Association (BSIA), and various content based retrieval systemsare described in a paper entitled: “Overview of Existing Content BasedVideo Retrieval Systems” by Shripad A. Bhat, Omkar V. Sardessai,Preetesh P. Kunde and Sarvesh S. Shirodkar of the Department ofElectronics and Telecommunication Engineering, Goa College ofEngineering, Farmagudi Ponda Goa, published February 2014 in ISSN No:2309-4893 International Journal of Advanced Engineering and GlobalTechnology Vol-2, Issue-2, which are both incorporated in their entiretyfor all purposes as if fully set forth herein.

Egomotion. Eegomotion is defined as the 3D motion of a camera within anenvironment, and typically refers to estimating a camera's motionrelative to a rigid scene. An example of egomotion estimation would beestimating a car's moving position relative to lines on the road orstreet signs being observed from the car itself. The estimation ofegomotion is important in autonomous robot navigation applications. Thegoal of estimating the egomotion of a camera is to determine the 3Dmotion of that camera within the environment using a sequence of imagestaken by the camera. The process of estimating a camera's motion withinan environment involves the use of visual odometry techniques on asequence of images captured by the moving camera. This is typically doneusing feature detection to construct an optical flow from two imageframes in a sequence generated from either single cameras or stereocameras. Using stereo image pairs for each frame helps reduce error andprovides additional depth and scale information.

Features are detected in the first frame, and then matched in the secondframe. This information is then used to make the optical flow field forthe detected features in those two images. The optical flow fieldillustrates how features diverge from a single point, the focus ofexpansion. The focus of expansion can be detected from the optical flowfield, indicating the direction of the motion of the camera, and thusproviding an estimate of the camera motion. There are other methods ofextracting egomotion information from images as well, including a methodthat avoids feature detection and optical flow fields and directly usesthe image intensities.

The computation of sensor motion from sets of displacement vectorsobtained from consecutive pairs of images is described in a paper byWilhelm Burger and Bir Bhanu entitled: “Estimating 3-D Egomotion fromPerspective Image Sequences”, published in IEEE TRANSACTIONS ON PATTERNANALYSIS AND MACHINE INTELLIGENCE, VOL. 12, NO. 11, NOVEMBER 1990, whichis incorporated in its entirety for all purposes as if fully set forthherein. The problem is investigated with emphasis on its application toautonomous robots and land vehicles. First, the effects of 3-D camerarotation and translation upon the observed image are discussed and inparticular the concept of the Focus Of Expansion (FOE). It is shown thatlocating the FOE precisely is difficult when displacement vectors arecorrupted by noise and errors. A more robust performance can be achievedby computing a 2-D region of possible FOE-locations (termed the fuzzyFOE) instead of looking for a single-point FOE. The shape of thisFOE-region is an explicit indicator for the accuracy of the result. Ithas been shown elsewhere that given the fuzzy FOE, a number of powerfulinferences about the 3-D scene structure and motion become possible.This paper concentrates on the aspects of computing the fuzzy FOE andshows the performance of a particular algorithm on real motion sequencestaken from a moving autonomous land vehicle.

Robust methods for estimating camera egomotion in noisy, real-worldmonocular image sequences in the general case of unknown observerrotation and translation with two views and a small baseline aredescribed in a paper by Andrew Jaegle, Stephen Phillips, and KostasDaniilidis of the University of Pennsylvania, Philadelphia, Pa., U.S.A.entitled: “Fast, Robust, Continuous Monocular Egomotion Computation”,downloaded from the Internet on January 2019, which is incorporated inits entirety for all purposes as if fully set forth herein. This is adifficult problem because of the nonconvex cost function of theperspective camera motion equation and because of non-Gaussian noisearising from noisy optical flow estimates and scene non-rigidity. Toaddress this problem, we introduce the expected residual likelihoodmethod (ERL), which estimates confidence weights for noisy optical flowdata using likelihood distributions of the residuals of the flow fieldunder a range of counterfactual model parameters. We show that ERL iseffective at identifying outliers and recovering appropriate confidenceweights in many settings. We compare ERL to a novel formulation of theperspective camera motion equation using a lifted kernel, a recentlyproposed optimization framework for joint parameter and confidenceweight estimation with good empirical properties. We incorporate thesestrategies into a motion estimation pipeline that avoids falling intolocal minima. We find that ERL outperforms the lifted kernel method andbaseline monocular egomotion estimation strategies on the challengingKITTI dataset, while adding almost no runtime cost over baselineegomotion methods.

Six algorithms for computing egomotion from image velocities aredescribed and evaluated in a paper by Tina Y. Tian, Carlo Tomasi, andDavid J. Heeger of the Department of Psychology and Computer ScienceDepartment of Stanford University, Stanford, Calif. 94305, entitled:“Comparison of Approaches to Egomotion Computation”, downloaded from theInternet on January 2019, which is incorporated in its entirety for allpurposes as if fully set forth herein. Various benchmarks areestablished for quantifying bias and sensitivity to noise, and forquantifying the convergence properties of those algorithms that requirenumerical search. The simulation results reveal some interesting andsurprising results. First, it is often written in the literature thatthe egomotion problem is difficult because translation (e.g., along theX-axis) and rotation (e.g., about the Y-axis) produce similar imagevelocities. It was found, to the contrary, that the bias and sensitivityof our six algorithms are totally invariant with respect to the axis ofrotation. Second, it is also believed by some that fixating helps tomake the egomotion problem easier. It was found, to the contrary, thatfixating does not help when the noise is independent of the imagevelocities. Fixation does help if the noise is proportional to speed,but this is only for the trivial reason that the speeds are slower underfixation. Third, it is widely believed that increasing the field of viewwill yield better performance, and it was found, to the contrary, thatthis is not necessarily true.

A system for estimating ego-motion of a moving camera for detection ofindependent moving objects in a scene is described in U.S. Pat. No.10,089,549 to Cao et al. entitled: “Valley search method for estimatingego-motion of a camera from videos”, which is incorporated in itsentirety for all purposes as if fully set forth herein. For consecutiveframes in a video captured by a moving camera, a first ego-translationestimate is determined between the consecutive frames from a first localminimum. From a second local minimum, a second ego-translation estimateis determined. If the first ego-translation estimate is equivalent tothe second ego-translation estimate, the second ego-translation estimateis output as the optimal solution. Otherwise, a cost function isminimized to determine an optimal translation until the firstego-translation estimate is equivalent to the second ego-translationestimate, and an optimal solution is output. Ego-motion of the camera isestimated using the optimal solution, and independent moving objects aredetected in the scene.

A system for compensating for ego-motion during video processing isdescribed in U.S. Patent Application Publication No. 2018/0225833 to Caoet al. entitled: “Efficient hybrid method for ego-motion from videoscaptured using an aerial camera”, which is incorporated in its entiretyfor all purposes as if fully set forth herein. The system generates aninitial estimate of camera ego-motion of a moving camera for consecutiveimage frame pairs of a video of a scene using a projected correlationmethod, the camera configured to capture the video from a movingplatform. An optimal estimation of camera ego-motion is generated usingthe initial estimate as an input to a valley search method or analternate line search method. All independent moving objects aredetected in the scene using the described hybrid method at superiorperformance compared to existing methods while saving computationalcost.

A method for estimating ego motion of an object moving on a surface isdescribed in U.S. Patent Application Publication No. 2015/0086078 toSibiryakov entitled: “Method for estimating ego motion of an object”,which is incorporated in its entirety for all purposes as if fully setforth herein. The method including generating at least two composite topview images of the surface on the basis of video frames provided by atleast one onboard video camera of the object moving on the surface;performing a region matching between consecutive top view images toextract global motion parameters of the moving object; calculating theego motion of the moving object from the extracted global motionparameters of the moving object.

Thermal camera. Thermal imaging is a method of improving visibility ofobjects in a dark environment by detecting the objects infraredradiation and creating an image based on that information. Thermalimaging, near-infrared illumination, and low-light imaging are the threemost commonly used night vision technologies. Unlike the other twomethods, thermal imaging works in environments without any ambient lightLike near-infrared illumination, thermal imaging can penetrateobscurants such as smoke, fog and haze. All objects emit infrared energy(heat) as a function of their temperature, and the infrared energyemitted by an object is known as its heat signature. In general, thehotter an object is, the more radiation it emits. A thermal imager (alsoknown as a thermal camera) is essentially a heat sensor that is capableof detecting tiny differences in temperature. The device collects theinfrared radiation from objects in the scene and creates an electronicimage based on information about the temperature differences. Becauseobjects are rarely precisely the same temperature as other objectsaround them, a thermal camera can detect them and they will appear asdistinct in a thermal image.

A thermal camera, also known as thermographic camera, is a device thatforms a heat zone image using infrared radiation, similar to a commoncamera that forms an image using visible light. Instead of the 400-700nanometer range of the visible light camera, infrared cameras operate inwavelengths as long as 14,000 nm (14 μm). A major difference fromoptical cameras is that the focusing lenses cannot be made of glass, asglass blocks long-wave infrared light. Typically the spectral range ofthermal radiation is from 7 to 14 mkm. Special materials such asGermanium, calcium fluoride, crystalline silicon or newly developedspecial type of Chalcogenide glass must be used. Except for calciumfluoride all these materials are quite hard but have high refractiveindex (n=4 for germanium) which leads to very high Fresnel reflectionfrom uncoated surfaces (up to more than 30%). For this reason most ofthe lenses for thermal cameras have antireflective coatings.

LIDAR. Light Detection And Ranging—LIDAR—also known as Lidar, LiDAR orLADAR (sometimes Light Imaging, Detection, And Ranging), is a surveyingtechnology that measures distance by illuminating a target with a laserlight. Lidar is popularly used as a technology to make high-resolutionmaps, with applications in geodesy, geomatics, archaeology, geography,geology, geomorphology, seismology, forestry, atmospheric physics,Airborne Laser Swath Mapping (ALSM) and laser altimetry, as well aslaser scanning or 3D scanning, with terrestrial, airborne and mobileapplications. Lidar typically uses ultraviolet, visible, or nearinfrared light to image objects. It can target a wide range ofmaterials, including non-metallic objects, rocks, rain, chemicalcompounds, aerosols, clouds and even single molecules. A narrowlaser-beam can map physical features with very high resolutions; forexample, an aircraft can map terrain at 30 cm resolution or better.Wavelengths vary to suit the target: from about 10 micrometers to the UV(approximately 250 nm). Typically light is reflected via backscattering.Different types of scattering are used for different LIDAR applications:most commonly Rayleigh scattering, Mie scattering, Raman scattering, andfluorescence. Based on different kinds of backscattering, the LIDAR canbe accordingly called Rayleigh Lidar, Mie Lidar, Raman Lidar, Na/Fe/KFluorescence Lidar, and so on. Suitable combinations of wavelengths canallow for remote mapping of atmospheric contents by identifyingwavelength-dependent changes in the intensity of the returned signal.Lidar has a wide range of applications, which can be divided intoairborne and terrestrial types. These different types of applicationsrequire scanners with varying specifications based on the data'spurpose, the size of the area to be captured, the range of measurementdesired, the cost of equipment, and more.

Airborne LIDAR (also airborne laser scanning) is when a laser scanner,while attached to a plane during flight, creates a 3D point cloud modelof the landscape. This is currently the most detailed and accuratemethod of creating digital elevation models, replacing photogrammetry.One major advantage in comparison with photogrammetry is the ability tofilter out vegetation from the point cloud model to create a digitalsurface model where areas covered by vegetation can be visualized,including rivers, paths, cultural heritage sites, etc. Within thecategory of airborne LIDAR, there is sometimes a distinction madebetween high-altitude and low-altitude applications, but the maindifference is a reduction in both accuracy and point density of dataacquired at higher altitudes. Airborne LIDAR may also be used to createbathymetric models in shallow water. Drones are being used with laserscanners, as well as other remote sensors, as a more economical methodto scan smaller areas. The possibility of drone remote sensing alsoeliminates any danger that crews of a manned aircraft may be subjectedto in difficult terrain or remote areas. Airborne LIDAR sensors are usedby companies in the remote sensing field. They can be used to create aDTM (Digital Terrain Model) or DEM (Digital Elevation Model); this isquite a common practice for larger areas as a plane can acquire 3-4 kmwide swaths in a single flyover. Greater vertical accuracy of below 50mm may be achieved with a lower flyover, even in forests, where it isable to give the height of the canopy as well as the ground elevation.Typically, a GNSS receiver configured over a georeferenced control pointis needed to link the data in with the WGS (World Geodetic System).

Terrestrial applications of LIDAR (also terrestrial laser scanning)happen on the Earth's surface and may be stationary or mobile.Stationary terrestrial scanning is most common as a survey method, forexample in conventional topography, monitoring, cultural heritagedocumentation and forensics. The 3D point clouds acquired from thesetypes of scanners can be matched with digital images taken of thescanned area from the scanner's location to create realistic looking 3Dmodels in a relatively short time when compared to other technologies.Each point in the point cloud is given the colour of the pixel from theimage taken located at the same angle as the laser beam that created thepoint.

Mobile LIDAR (also mobile laser scanning) is when two or more scannersare attached to a moving vehicle to collect data along a path. Thesescanners are almost always paired with other kinds of equipment,including GNSS receivers and IMUs. One example application is surveyingstreets, where power lines, exact bridge heights, bordering trees, etc.all need to be taken into account. Instead of collecting each of thesemeasurements individually in the field with a tachymeter, a 3D modelfrom a point cloud can be created where all of the measurements neededcan be made, depending on the quality of the data collected. Thiseliminates the problem of forgetting to take a measurement, so long asthe model is available, reliable and has an appropriate level ofaccuracy.

Autonomous vehicles use LIDAR for obstacle detection and avoidance tonavigate safely through environments. Cost map or point cloud outputsfrom the LIDAR sensor provide the necessary data for robot software todetermine where potential obstacles exist in the environment and wherethe robot is in relation to those potential obstacles. LIDAR sensors arecommonly used in robotics or vehicle automation. The very firstgenerations of automotive adaptive cruise control systems used onlyLIDAR sensors.

LIDAR technology is being used in robotics for the perception of theenvironment as well as object classification. The ability of LIDARtechnology to provide three-dimensional elevation maps of the terrain,high precision distance to the ground, and approach velocity can enablesafe landing of robotic and manned vehicles with a high degree ofprecision. LiDAR has been used in the railroad industry to generateasset health reports for asset management and by departments oftransportation to assess their road conditions. LIDAR is used inAdaptive Cruise Control (ACC) systems for automobiles. Systems use aLIDAR device mounted on the front of the vehicle, such as the bumper, tomonitor the distance between the vehicle and any vehicle in front of it.In the event the vehicle in front slows down or is too close, the ACCapplies the brakes to slow the vehicle. When the road ahead is clear,the ACC allows the vehicle to accelerate to a speed preset by thedriver. Any apparatus herein, which may be any of the systems, devices,modules, or functionalities described herein, may be integrated with, orused for, Light Detection And Ranging (LIDAR), such as airborne,terrestrial, automotive, or mobile LIDAR.

Time-frequency Analysis. A time-frequency analysis comprises thosetechniques that study a signal in both the time and frequency domainssimultaneously, using various time-frequency representations. Ratherthan viewing a 1-dimensional signal (a function, real or complex-valued,whose domain is the real line) and some transform (another functionwhose domain is the real line, obtained from the original via sometransform), time-frequency analysis studies a two-dimensional signal—afunction whose domain is the two-dimensional real plane, obtained fromthe signal via a time-frequency transform. Time-Frequency analysis isdescribed in an article by Rolf Hut (September 2004) entitled: “TimeFrequency Analysis—a Comparison between cochlear modeling and existingmethods”, and in an article by Franz Hlawatsch and Gerald Matz (of theInstitute of Communications and radio-Frequency Engineering, ViennaUniversity of Technology) entitled: “Time-Frequency Signal Processing: AStatistical Perspective”, which are both incorporated in their entiretyfor all purposes as if fully set forth herein. One of the most basicforms of time-frequency analysis is the Short-Time Fourier Transform(STFT), but more sophisticated techniques have been developed, such aswavelets.

There are several different ways to formulate a valid time-frequencydistribution function, resulting in several well-known time-frequencydistributions, such as: Short-time Fourier transform (including theGabor transform); Wavelet transform; Bilinear time-frequencydistribution function (Wigner distribution function, or WDF); andModified Wigner distribution function or Gabor-Wigner distributionfunction.

Pitch/Roll/Yaw (Spatial orientation and motion). Any device that canmove in space, such as an aircraft in flight, is typically free torotate in three dimensions: yaw-nose left or right about an axis runningup and down; pitch-nose up or down about an axis running from wing towing; and roll-rotation about an axis running from nose to tail, aspictorially shown in FIG. 2. The axes are alternatively designated asvertical, transverse, and longitudinal respectively. These axes movewith the vehicle and rotate relative to the Earth along with the craft.These rotations are produced by torques (or moments) about the principalaxes. On an aircraft, these are intentionally produced by means ofmoving control surfaces, which vary the distribution of the netaerodynamic force about the vehicle's center of gravity. Elevators(moving flaps on the horizontal tail) produce pitch, a rudder on thevertical tail produces yaw, and ailerons (flaps on the wings that movein opposing directions) produce roll. On a spacecraft, the moments areusually produced by a reaction control system consisting of small rocketthrusters used to apply asymmetrical thrust on the vehicle. Normal axis,or yaw axis, is an axis drawn from top to bottom, and perpendicular tothe other two axes. Parallel to the fuselage station. Transverse axis,lateral axis, or pitch axis, is an axis running from the pilot's left toright in piloted aircraft, and parallel to the wings of a wingedaircraft. Parallel to the buttock line. Longitudinal axis, or roll axis,is an axis drawn through the body of the vehicle from tail to nose inthe normal direction of flight, or the direction the pilot faces.Parallel to the waterline.

Vertical axis (yaw)—The yaw axis has its origin at the center of gravityand is directed towards the bottom of the aircraft, perpendicular to thewings and to the fuselage reference line. Motion about this axis iscalled yaw. A positive yawing motion moves the nose of the aircraft tothe right. The rudder is the primary control of yaw. Transverse axis(pitch)—The pitch axis (also called transverse or lateral axis) has itsorigin at the center of gravity and is directed to the right, parallelto a line drawn from wingtip to wingtip. Motion about this axis iscalled pitch. A positive pitching motion raises the nose of the aircraftand lowers the tail. The elevators are the primary control of pitch.Longitudinal axis (roll)—The roll axis (or longitudinal axis) has itsorigin at the center of gravity and is directed forward, parallel to thefuselage reference line. Motion about this axis is called roll. Anangular displacement about this axis is called bank. A positive rollingmotion lifts the left wing and lowers the right wing. The pilot rolls byincreasing the lift on one wing and decreasing it on the other. Thischanges the bank angle. The ailerons are the primary control of bank.

PTZ. A Pan-Tilt-Zoom camera (PTZ camera) is a camera that is capable ofremote directional and zoom control. PTZ is an abbreviation for pan,tilt and zoom and reflects the movement options of the camera. Othertypes of cameras are ePTZ or virtual pan-tilt-zoom (VPTZ) where ahigh-resolution camera digitally zooms and pans into portions of theimage, with no physical camera movement. Ultra-low bandwidthsurveillance streaming technologies use VPTZ to stream user-definedareas in higher quality without increasing overall bandwidth usage.Surveillance cameras of this type are often connected to a digital videorecorder which records the full field of view in full quality. PTZCameras are commonly used in applications such as surveillance, videoconferencing, live production, lecture capture and distance learning.

Recent PTZ cameras include a built-in firmware program that monitors thechange of pixels generated by the video clip in the camera. When thepixels change due to movement within the camera's field of view, thecamera can actually focus on the pixel variation and move the camera inan attempt to center the pixel fluctuation on the video chip. Thisresults in the camera following movement. The program allows the camerato estimate the size of the object which is moving and distance of themovement from the camera. With this estimate, the camera can adjust thecamera's optical lens, zooming in and out, in an attempt to stabilizethe size of pixel fluctuation as a percentage of total viewing area.Once the movement exits the camera's field of view, the camera returnsto a pre-programmed or “parked” position until it senses pixel variationand the process starts over again.

Level meter. A spirit level, bubble level or simply a level is aninstrument designed to indicate whether a surface is horizontal (level)or vertical (plumb). Different types of spirit levels may be used bycarpenters, stonemasons, bricklayers, other building trades workers,surveyors, millwrights and other metalworkers, and in some photographicor videographic work, and typically involves a sealed glass tubecontaining alcohol and an air bubble. Early spirit levels had veryslightly curved glass vials with constant inner diameter at each viewingpoint. These vials are incompletely filled with a liquid, usually acolored spirit or alcohol, leaving a bubble in the tube. They have aslight upward curve, so that the bubble naturally rests in the center,the highest point. At slight inclinations, the bubble travels away fromthe marked center position. Where a spirit level must also be usableupside-down or on its side, the curved constant-diameter tube isreplaced by an uncurved barrel-shaped tube with a slightly largerdiameter in its middle.

Alcohols such as ethanol are often used rather than water, sincealcohols have low viscosity and surface tension, which allows the bubbleto travel the tube quickly and settle accurately with minimalinterference with the glass surface. Alcohols also have a much widerliquid temperature range, and are less susceptible to break the vial aswater could due to ice expansion. A colorant such as fluorescein,typically yellow or green, may be added to increase the visibility ofthe bubble. An extension of the spirit level is the bull's eye level: acircular, flat-bottomed device with the liquid under a slightly convexglass face with a circle at the center. It serves to level a surfaceacross a plane, while the tubular level only does so in the direction ofthe tube.

Tilting level, dumpy level, or automatic level are terms used to referto types of leveling instruments as used in surveying to measure heightdifferences over larger distances. It has a spirit level mounted on atelescope (perhaps 30 power) with cross-hairs, itself mounted on atripod. The observer reads height values off two graduated verticalrods, one ‘behind’ and one ‘in front’, to obtain the height differencebetween the ground points on which the rods are resting. Starting from apoint with a known elevation and going cross country (successive pointsbeing perhaps 100 meters (328 ft) apart) height differences can bemeasured cumulatively over long distances and elevations can becalculated. Precise leveling is supposed to give the difference inelevation between two points one kilometer (0.62 miles) apart correct towithin a few millimeters.

A traditional carpenter's spirit level looks like a short plank of woodand often has a wide body to ensure stability, and that the surface isbeing measured correctly. In the middle of the spirit level is a smallwindow where the bubble and the tube is mounted. Two notches (or rings)designate where the bubble should be if the surface is levelled. Oftenan indicator for a 45 degree inclination is included. A line level is alevel designed to hang on a builder's string line. The body of the levelincorporates small hooks to allow it to attach and hang from the stringline. The body is lightweight, so as not to weigh down the string line,it is also small in size as the string line in effect becomes the body;when the level is hung in the center of the string, each leg of thestring line extends the levels plane.

Digital levels are increasingly common in replacing conventional spiritlevels particularly in civil engineering applications, such as buildingconstruction and steel structure erection, for on-site angle alignmentand leveling tasks. The industry practitioners often refer thoseleveling tool as “construction level”, “heavy duty level”,“inclinometer”, or “protractor”. These modern electronic levels are (i)capable of displaying precise numeric angles within 360° with highaccuracy, (ii) digital readings can be read from a distance withclarity, (iii) affordable price resulted from mass adoption, providingadvantages that the traditional levels are unable to match. Typically,these features enable steel beam frames under construction to beprecisely aligned and levelled to the required orientation, which isvital to effectively ensure the stability, strength, and rigidity ofsteel structures on sites. Digital levels, embedded with angular MEMStechnology effectively improve productivity and quality of many moderncivil structures used by on-site constructions workers. Some of therecent models are even designed with waterproof IP65 and impactresistance features to meet the stringent working environment of theindustry.

Inclinometer. An inclinometer or clinometer is an instrument formeasuring angles of slope (or tilt), elevation or depression of anobject with respect to gravity. It is also known as a tilt meter, tiltindicator, slope alert, slope gauge, gradient meter, gradiometer, levelgauge, level meter, declinometer, and pitch & roll indicator.Clinometers measure both inclines (positive slopes, as seen by anobserver looking upwards) and declines (negative slopes, as seen by anobserver looking downward) using three different units of measure:degrees, percent, and topo. Astrolabes are inclinometers that were usedfor navigation and locating astronomical objects from ancient times tothe Renaissance.

Tilt sensors and inclinometers generate an artificial horizon andmeasure angular tilt with respect to this horizon. They are used incameras, aircraft flight controls, automobile security systems, andspecialty switches and are also used for platform leveling, boom angleindication, and in other applications requiring measurement of tilt.Common implementations of tilt sensors and inclinometers areaccelerometer, Liquid Capacitive, electrolytic, gas bubble in liquid,and pendulum.

Traditional spirit levels and pendulum-based electronic levelinginstruments are usually constrained by only single-axis and narrow tiltmeasurement range. However, most precision leveling, angle measurement,alignment and surface flatness profiling tasks essentially involve a2-dimensional surface plane angle rather than two independent orthogonalsingle-axis objects. 2-Axis inclinometers that are built with MEMS tiltsensors provides simultaneous 2-dimensional angle readings of a surfaceplane tangent to earth datum.

2-Axis Digital Inclinometer. 2-axis MEMS technology enables simultaneoustwo-dimensional (X-Y plane) tilt angles (i.e. pitch & roll) measurement,eliminates tedious trial-and-error (i.e. going back-and-forth)experienced when using single-axis levels to adjust machine footings toattain a precise leveling position. 2-axis MEMS inclinometers can bedigitally compensated and precisely calibrated for non-linearity foroperating temperature variation resulting in higher angular accuracyover wider angular measurement range. 2-axis MEMS inclinometer withbuilt-in accelerometer sensors may generate numerical data tabulated inthe form of vibration profiles that enable machine installer to trackand assess alignment quality in real-time and verify structurepositional stability by comparing machine's leveling profiles before andafter setting up.

Gyroscope. A gyroscope is a device commonly used for measuring ormaintaining orientation and angular velocity. It is typically based on aspinning wheel or disc in which the axis of rotation is free to assumeany orientation by itself. When rotating, the orientation of this axisis unaffected by tilting or rotation of the mounting, according to theconservation of angular momentum. Gyroscopes based on other operatingprinciples also exist, such as the microchip-packaged MEMS gyroscopesfound in electronic devices, solid-state ring lasers, fibre-opticgyroscopes, and the extremely sensitive quantum gyroscope. MEMSgyroscopes are popular in some consumer electronics, such assmartphones.

A gyroscope is typically a wheel mounted in two or three gimbals, whichare pivoted supports that allow the rotation of the wheel about a singleaxis. A set of three gimbals, one mounted on the other with orthogonalpivot axes, may be used to allow a wheel mounted on the innermost gimbalto have an orientation remaining independent of the orientation, inspace, of its support. In the case of a gyroscope with two gimbals, theouter gimbal, which is the gyroscope frame, is mounted so as to pivotabout an axis in its own plane determined by the support. This outergimbal possesses one degree of rotational freedom and its axis possessesnone. The inner gimbal is mounted in the gyroscope frame (outer gimbal)so as to pivot about an axis in its own plane that is alwaysperpendicular to the pivotal axis of the gyroscope frame (outer gimbal).This inner gimbal has two degrees of rotational freedom. The axle of thespinning wheel defines the spin axis. The rotor is constrained to spinabout an axis, which is always perpendicular to the axis of the innergimbal. So the rotor possesses three degrees of rotational freedom andits axis possesses two. The wheel responds to a force applied to theinput axis by a reaction force to the output axis. A gyroscope flywheelwill roll or resist about the output axis depending upon whether theoutput gimbals are of a free or fixed configuration. Examples of somefree-output-gimbal devices would be the attitude reference gyroscopesused to sense or measure the pitch, roll and yaw attitude angles in aspacecraft or aircraft.

Accelerometer. An accelerometer is a device that measures properacceleration, typically being the acceleration (or rate of change ofvelocity) of a body in its own instantaneous rest frame. Single- andmulti-axis models of accelerometer are available to detect magnitude anddirection of the proper acceleration, as a vector quantity, and can beused to sense orientation (because direction of weight changes),coordinate acceleration, vibration, shock, and falling in a resistivemedium (a case where the proper acceleration changes, since it starts atzero, then increases). Micro-machined Microelectromechanical Systems(MEMS) accelerometers are increasingly present in portable electronicdevices and video game controllers, to detect the position of the deviceor provide for game input. Conceptually, an accelerometer behaves as adamped mass on a spring. When the accelerometer experiences anacceleration, the mass is displaced to the point that the spring is ableto accelerate the mass at the same rate as the casing. The displacementis then measured to give the acceleration.

In commercial devices, piezoelectric, piezoresistive and capacitivecomponents are commonly used to convert the mechanical motion into anelectrical signal. Piezoelectric accelerometers rely on piezoceramics(e.g. lead zirconate titanate) or single crystals (e.g. quartz,tourmaline). They are unmatched in terms of their upper frequency range,low packaged weight and high temperature range. Piezoresistiveaccelerometers are preferred in high shock applications. Capacitiveaccelerometers typically use a silicon micro-machined sensing element.Their performance is superior in the low frequency range and they can beoperated in servo mode to achieve high stability and linearity. Modernaccelerometers are often small micro electro-mechanical systems (MEMS),and are indeed the simplest MEMS devices possible, consisting of littlemore than a cantilever beam with a proof mass (also known as seismicmass). Damping results from the residual gas sealed in the device. Aslong as the Q-factor is not too low, damping does not result in a lowersensitivity. Most micromechanical accelerometers operate in-plane, thatis, they are designed to be sensitive only to a direction in the planeof the die. By integrating two devices perpendicularly on a single die atwo-axis accelerometer can be made. By adding another out-of-planedevice, three axes can be measured. Such a combination may have muchlower misalignment error than three discrete models combined afterpackaging.

A laser accelerometer comprises a frame having three orthogonal inputaxes and multiple proof masses, each proof mass having a predeterminedblanking surface. A flexible beam supports each proof mass. The flexiblebeam permits movement of the proof mass on the input axis. A laser lightsource provides a light ray. The laser source is characterized to have atransverse field characteristic having a central null intensity region.A mirror transmits a ray of light to a detector. The detector ispositioned to be centered to the light ray and responds to thetransmitted light ray intensity to provide an intensity signal. Theintensity signal is characterized to have a magnitude related to theintensity of the transmitted light ray. The proof mass blanking surfaceis centrally positioned within and normal to the light ray nullintensity region to provide increased blanking of the light ray inresponse to transverse movement of the mass on the input axis. The proofmass deflects the flexible beam and moves the blanking surface in adirection transverse to the light ray to partially blank the light beamin response to acceleration in the direction of the input axis. Acontrol responds to the intensity signal to apply a restoring force torestore the proof mass to a central position and provides an outputsignal proportional to the restoring force.

A motion sensor may include one or more accelerometers, which measuresthe absolute acceleration or the acceleration relative to freefall. Forexample, one single-axis accelerometer per axis may be used, requiringthree such accelerometers for three-axis sensing. The motion sensor maybe a single or multi-axis sensor, detecting the magnitude and directionof the acceleration as a vector quantity, and thus can be used to senseorientation, acceleration, vibration, shock and falling. The motionsensor output may be analog or digital signals, representing themeasured values. The motion sensor may be based on a piezoelectricaccelerometer that utilizes the piezoelectric effect of certainmaterials to measure dynamic changes in mechanical variables (e.g.,acceleration, vibration, and mechanical shock). Piezoelectricaccelerometers commonly rely on piezoceramics (e.g., lead zirconatetitanate) or single crystals (e.g., Quartz, Tourmaline). A piezoelectricquartz accelerometer is disclosed in U.S. Pat. No. 7,716,985 to Zhang etal. entitled: “Piezoelectric Quartz Accelerometer”, U.S. Pat. No.5,578,755 to Offenberg entitled: “Accelerometer Sensor of CrystallineMaterial and Method for Manufacturing the Same” and U.S. Pat. No.5,962,786 to Le Traon et al. entitled: “Monolithic AccelerometricTransducer”, which are all incorporated in their entirety for allpurposes as if fully set forth herein. Alternatively or in addition, themotion sensor may be based on the Micro Electro-Mechanical Systems(MEMS, a.k.a. Micro-mechanical electrical system) technology. A MEMSbased motion sensor is disclosed in U.S. Pat. No. 7,617,729 to Axelrodet al. entitled: “Accelerometer”, U.S. Pat. No. 6,670,212 to McNie etal. entitled: “Micro-Machining” and in U.S. Pat. No. 7,892,876 toMehregany entitled: “Three-axis Accelerometers and Fabrication Methods”,which are all incorporated in their entirety for all purposes as iffully set forth herein. An example of MEMS motion sensor is LIS302DLmanufactured by STMicroelectronics NV and described in Data-sheetLIS302DL STMicroelectronics NV, ‘MEMS motion sensor 3-axis-±2 g/±8 gsmart digital output “piccolo” accelerometer’, Rev. 4, October 2008,which is incorporated in its entirety for all purposes as if fully setforth herein.

Alternatively or in addition, the motion sensor may be based onelectrical tilt and vibration switch or any other electromechanicalswitch, such as the sensor described in U.S. Pat. No. 7,326,866 toWhitmore et al. entitled: “Omnidirectional Tilt and vibration sensor”,which is incorporated in its entirety for all purposes as if fully setforth herein. An example of an electromechanical switch is SQ-SEN-200available from SignalQuest, Inc. of Lebanon, N.H., USA, described in thedata-sheet ‘DATASHEET SQ-SEN-200 Omnidirectional Tilt and VibrationSensor’ Updated 2009-08-03, which is incorporated in its entirety forall purposes as if fully set forth herein. Other types of motion sensorsmay be equally used, such as devices based on piezoelectric,piezo-resistive, and capacitive components, to convert the mechanicalmotion into an electrical signal. Using an accelerometer to control isdisclosed in U.S. Pat. No. 7,774,155 to Sato et al. entitled:“Accelerometer-Based Controller”, which is incorporated in its entiretyfor all purposes as if fully set forth herein.

IMU. The Inertial Measurement Unity (IMU) is an integrated sensorpackage that combines multiple accelerometers and gyros to produce athree dimensional measurement of both specific force and angular rate,with respect to an inertial reference frame, as for example theEarth-Centered Inertial (ECI) reference frame. Specific force is ameasure of acceleration relative to free-fall. Subtracting thegravitational acceleration results in a measurement of actual coordinateacceleration. Angular rate is a measure of rate of rotation. Typically,IMU includes the combination of only a 3-axis accelerometer combinedwith a 3-axis gyro. An onboard processor, memory, and temperature sensormay be included to provide a digital interface, unit conversion and toapply a sensor calibration model. An IMU may include one or more motionsensors.

An Inertial Measurement Unit (IMU) further measures and reports a body'sspecific force, angular rate, and sometimes the magnetic fieldsurrounding the body, using a combination of accelerometers andgyroscopes, sometimes also magnetometers. IMUs are typically used tomaneuver aircraft, including unmanned aerial vehicles (UAVs), among manyothers, and spacecraft, including satellites and landers. The IMU is themain component of inertial navigation systems used in aircraft,spacecraft, watercraft, drones, UAV and guided missiles among others. Inthis capacity, the data collected from the IMU's sensors allows acomputer to track a craft's position, using a method known as deadreckoning.

An inertial measurement unit works by detecting the current rate ofacceleration using one or more accelerometers, and detects changes inrotational attributes like pitch, roll and yaw using one or moregyroscopes. Typical IMU also includes a magnetometer, mostly to assistcalibration against orientation drift. Inertial navigation systemscontain IMUs that have angular and linear accelerometers (for changes inposition); some IMUs include a gyroscopic element (for maintaining anabsolute angular reference). Angular accelerometers measure how thevehicle is rotating in space. Generally, there is at least one sensorfor each of the three axes: pitch (nose up and down), yaw (nose left andright) and roll (clockwise or counter-clockwise from the cockpit).Linear accelerometers measure non-gravitational accelerations of thevehicle. Since it can move in three axes (up & down, left & right,forward & back), there is a linear accelerometer for each axis. Thethree gyroscopes are commonly placed in a similar orthogonal pattern,measuring rotational position in reference to an arbitrarily chosencoordinate system. A computer continually calculates the vehicle'scurrent position. First, for each of the six degrees of freedom (x,y,z,and θx, θy, and θz), it integrates over time the sensed acceleration,together with an estimate of gravity, to calculate the current velocity.Then it integrates the velocity to calculate the current position.

An example for an IMU is a module Part Number LSM9DS1 available fromSTMicroelectronics NV headquartered in Geneva, Switzerland and describedin a datasheet published March 2015 and entitled: “LSM9DS1—iNEMOinertial module: 3D accelerometer, 3D gyroscope, 3D magnetometer”, whichis incorporated in its entirety for all purposes as if fully set forthherein. Another example for an IMU is unit Part Number STIM300 availablefrom Sensonor AS, headquartered in Horten, Norway, and is described in adatasheet dated October 2015 [TS1524 rev. 20] entitled:“ButterflyGyro™—STIM300 Intertia Measurement Unit”, which isincorporated in its entirety for all purposes as if fully set forthherein. Using IMU for human motion or positioning is described in aMaster's Thesis by Martin Veskrna of Masaryk University, Faculty ofInformatics dated 2013, entitled: “Positioning system for small devicesusing principles of inertial navigation system”, in an article by SamNaghshineh, Golafsoun Ameri, Mazdak Zereshki & Dr. S. Krishnan, Dr. M.Abdoli-Eramaki (downloaded from the Internet March 2016) entitled:“Human Motion capture using Tri-Axial accelerometers”, and in a paper byXiaoping Yun et al. published 2007 IEEE International Conference onRobotics and Automation (Rome, Italy, 10-14 April 2007) entitled:“Self-Contained Position Tracking of Human Movement Using SmallInertial/Magnetic Sensor Module”, which are all incorporated in theirentirety for all purposes as if fully set forth herein.

Streaming. Streaming media is multimedia that is constantly received byand presented to an end-user while being delivered by a provider. Aclient media player can begin playing the data (such as a movie) beforethe entire file has been transmitted. Distinguishing delivery methodfrom the media distributed applies specifically to telecommunicationsnetworks, as most of the delivery systems are either inherentlystreaming (e.g., radio, television), or inherently non-streaming (e.g.,books, video cassettes, audio CDs). Live streaming refers to contentdelivered live over the Internet, and requires a form of source media(e.g. a video camera, an audio interface, screen capture software), anencoder to digitize the content, a media publisher, and a contentdelivery network to distribute and deliver the content. Streamingcontent may be according to, compatible with, or based on, IETF RFC 2550entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETFRFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETFRFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”, which are allincorporated in their entirety for all purposes as if fully set forthherein. Video streaming is further described in a published 2002 paperby Hewlett-Packard Company (HP®) authored by John G. Apostolopoulos,Wai-Tian, and Susie J. Wee and entitled: “Video Streaming: Concepts,Algorithms, and Systems”, which is incorporated in its entirety for allpurposes as if fully set forth herein.

An audio stream may be compressed using an audio codec such as MP3,Vorbis or AAC, and a video stream may be compressed using a video codecsuch as H.264 or VP8. Encoded audio and video streams may be assembledin a container bitstream such as MP4, FLV, WebM, ASF or ISMA. Thebitstream is typically delivered from a streaming server to a streamingclient using a transport protocol, such as MMS or RTP. Newertechnologies such as HLS, Microsoft's Smooth Streaming, Adobe's HDS andfinally MPEG-DASH have emerged to enable adaptive bitrate (ABR)streaming over HTTP as an alternative to using proprietary transportprotocols. The streaming client may interact with the streaming serverusing a control protocol, such as MMS or RTSP.

Streaming media may use Datagram protocols, such as the User DatagramProtocol (UDP), where the media stream is sent as a series of smallpackets. However, there is no mechanism within the protocol to guaranteedelivery, so if data is lost, the stream may suffer a dropout. Otherprotocols may be used, such as the Real-time Streaming Protocol (RTSP),Real-time Transport Protocol (RTP) and the Real-time Transport ControlProtocol (RTCP). RTSP runs over a variety of transport protocols, whilethe latter two typically use UDP. Another approach is HTTP adaptivebitrate streaming that is based on HTTP progressive download, designedto incorporate both the advantages of using a standard web protocol, andthe ability to be used for streaming even live content is adaptivebitrate streaming. Reliable protocols, such as the Transmission ControlProtocol (TCP), guarantee correct delivery of each bit in the mediastream, using a system of timeouts and retries, which makes them morecomplex to implement. Unicast protocols send a separate copy of themedia stream from the server to each recipient, and are commonly usedfor most Internet connections.

Multicasting broadcasts the same copy of the multimedia over the entirenetwork to a group of clients, and may use multicast protocols that weredeveloped to reduce the server/network loads resulting from duplicatedata streams that occur when many recipients receive unicast contentstreams, independently. These protocols send a single stream from thesource to a group of recipients, and depending on the networkinfrastructure and type, the multicast transmission may or may not befeasible. IP Multicast provides the capability to send a single mediastream to a group of recipients on a computer network, and a multicastprotocol, usually Internet Group Management Protocol, is used to managedelivery of multicast streams to the groups of recipients on a LAN.Peer-to-peer (P2P) protocols arrange for prerecorded streams to be sentbetween computers, thus preventing the server and its networkconnections from becoming a bottleneck. HTTP Streaming—(a.k.a.Progressive Download; Streaming) allows for that while streaming contentis being downloaded, users can interact with, and/or view it. VODstreaming is further described in a NETFLIX® presentation dated May 2013by David Ronca, entitled: “A Brief History of Neflix Streaming”, whichis incorporated in its entirety for all purposes as if fully set forthherein.

Media streaming techniques are further described in a white paperpublished October 2005 by Envivio® and authored by Alex MacAulay, BorisFelts, and Yuval Fisher, entitled: “WHITEPAPER—IP Streaming of MPEG-4”Native RTP vs MPEG-2 Transport Stream”, in an overview published 2014 byApple Inc.—Developer, entitled: “HTTP Live Streaming Overview”, and in apaper by Thomas Stockhammer of Qualcomm Incorporated entitled: “DynamicAdaptive Streaming over HTTP—Design Principles and Standards”, in aMicrosoft Corporation published March 2009 paper authored by AlexZambelli and entitled: “IIS Smooth Streaming Technical Overview”, in anarticle by Liang Chen, Yipeng Zhou, and Dah Ming Chiu dated 10 Apr. 2014entitled: “Smart Streaming for Online Video Services”, in Celtic-Pluspublication (downloaded 2-2016 from the Internet) referred to as H2B2VSD1 1 1 State-of-the-art V2.0.docx' entitled: “H2B2VS D1.1.1 Report onthe state of the art technologies for hybrid distribution of TVservices”, and in a technology brief by Apple Computer, Inc. publishedMarch 2005 (Document No. L308280A) entitled: “QuickTime Streaming”,which are all incorporated in their entirety for all purposes as iffully set forth herein.

DSP. A Digital Signal Processor (DSP) is a specialized microprocessor(or a SIP block), with its architecture optimized for the operationalneeds of digital signal processing, serving the goal of DSPs is usuallyto measure, filter and/or compress continuous real-world analog signals.Most general-purpose microprocessors can also execute digital signalprocessing algorithms successfully, but dedicated DSPs usually havebetter power efficiency thus they are more suitable in portable devicessuch as mobile phones because of power consumption constraints. DSPsoften use special memory architectures that are able to fetch multipledata and/or instructions at the same time. Digital signal processingalgorithms typically require a large number of mathematical operationsto be performed quickly and repeatedly on a series of data samples.Signals (perhaps from audio or video sensors) are constantly convertedfrom analog to digital, manipulated digitally, and then converted backto analog form. Many DSP applications have constraints on latency; thatis, for the system to work, the DSP operation must be completed withinsome fixed time, and deferred (or batch) processing is not viable. Aspecialized digital signal processor, however, will tend to provide alower-cost solution, with better performance, lower latency, and norequirements for specialized cooling or large batteries. Thearchitecture of a digital signal processor is optimized specifically fordigital signal processing. Most also support some of the features as anapplications processor or microcontroller, since signal processing israrely the only task of a system. Some useful features for optimizingDSP algorithms are outlined below.

Hardware features visible through DSP instruction sets commonly includehardware modulo addressing, allowing circular buffers to be implementedwithout having to constantly test for wrapping; a memory architecturedesigned for streaming data, using DMA extensively and expecting code tobe written to know about cache hierarchies and the associated delays;driving multiple arithmetic units may require memory architectures tosupport several accesses per instruction cycle; separate program anddata memories (Harvard architecture), and sometimes concurrent access onmultiple data buses; and special SIMD (single instruction, multipledata) operations. Digital signal processing is further described in abook by John G. Proakis and Dimitris G. Manolakis, published 1996 byPrentice-Hall Inc. [ISBN 0-13-394338-9] entitled: “Third Edition—DIGITALSIGNAL PROCESSING—Principles, Algorithms, and Application”, and in abook by Steven W. Smith entitled: “The Scientist and Engineer's Guideto—Digital Signal Processing—Second Edition”, published by CaliforniaTechnical Publishing [ISBN 0-9960176-7-6], which are both incorporatedin their entirety for all purposes as if fully set forth herein.

Convolution and correlation. A convolution is a mathematical operationon two signals or functions (f and g) to produce a third function (orsignal) that expresses how the shape of one is modified by the other.The term convolution refers to both the result function and to theprocess of computing it. For real-valued functions, of a continuous ordiscrete variable, it differs from cross-correlation only in that eitherf (x) or g(x) is reflected about the y-axis; thus it is across-correlation of f(x) and g(−x), or f (−x) and g(x). For continuousfunctions, the cross-correlation operator is the adjoint of theconvolution operator. Computing the inverse of the convolution operationis known as deconvolution. Correlation is a measure of similaritybetween two signals. There are several correlation coefficients formeasuring the degree of correlation. The most common of these is thePearson correlation coefficient, which is sensitive only to a linearrelationship between two variables (which may be present even when onevariable is a nonlinear function of the other). Other correlationcoefficients have been developed to be more robust than the Pearsoncorrelation, and are more sensitive to nonlinear relationships.Correlation operation is typically carried out similar to convolution,except for left-for-right flip of one signal, and is commonly used tooptimally detect a known known waveform in a signal.

Cross-correlation is a measure of similarity of two series as a functionof the displacement of one relative to the other, also known as asliding dot product or sliding inner-product. It is commonly used forsearching a long signal for a shorter, known feature, and hasapplications in pattern recognition, single particle analysis, electrontomography, averaging, cryptanalysis, and neurophysiology. Thecross-correlation is similar in nature to the convolution of twofunctions. In an autocorrelation, which is the cross-correlation of asignal with itself, there will always be a peak at a lag of zero, andits size will be the signal energy.

Convolution and correlation are described in Chapter 2 entitled:“Convolution and Correlation” of the book “Analog and Digital Signalsand Systems” by Yarlagadda, R. K. R. published 2010 by SpringerScience+Business Media [ISBN: 978-1-4419-0033-3], which is incorporatedin its entirety for all purposes as if fully set forth herein. Apractical example of cross-correlation function is the function ‘xcorr’of MATLAB provided by The MathWorks®, Inc. and described in the webpagehttps://www.mathworks.com/help/signal/ref/xcorr.html downloaded January2019, which is incorporated in its entirety for all purposes as if fullyset forth herein.

Smartphone. A mobile phone (also known as a cellular phone, cell phone,smartphone, or hand phone) is a device which can make and receivetelephone calls over a radio link whilst moving around a wide geographicarea, by connecting to a cellular network provided by a mobile networkoperator. The calls are to and from the public telephone network, whichincludes other mobiles and fixed-line phones across the world. TheSmartphones are typically hand-held and may combine the functions of apersonal digital assistant (PDA), and may serve as portable mediaplayers and camera phones with high-resolution touch-screens, webbrowsers that can access, and properly display, standard web pagesrather than just mobile-optimized sites, GPS navigation, Wi-Fi andmobile broadband access. In addition to telephony, the Smartphones maysupport a wide variety of other services such as text messaging, MMS,email, Internet access, short-range wireless communications (infrared,Bluetooth), business applications, gaming and photography.

An example of a contemporary smartphone is model iPhone 6 available fromApple Inc., headquartered in Cupertino, Calif., U.S.A. and described iniPhone 6 technical specification (retrieved October 2015 fromwww.apple.com/iphone-6/specs/), and in a User Guide dated 2015(019-00155/2015-06) by Apple Inc. entitled: “iPhone User Guide For iOS8.4 Software”, which are both incorporated in their entirety for allpurposes as if fully set forth herein. Another example of a smartphoneis Samsung Galaxy S6 available from Samsung Electronics headquartered inSuwon, South-Korea, described in the user manual numbered English (EU),March 2015 (Rev. 1.0) entitled: “SM-G925F SM-G925FQ SM-G9251 UserManual” and having features and specification described in “Galaxy S6Edge—Technical Specification” (retrieved October 2015 fromwww.samsung.com/us/explore/galaxy-s-6-features-and-specs), which areboth incorporated in their entirety for all purposes as if fully setforth herein.

A mobile operating system (also referred to as mobile OS), is anoperating system that operates a smartphone, tablet, PDA, or anothermobile device. Modern mobile operating systems combine the features of apersonal computer operating system with other features, including atouchscreen, cellular, Bluetooth, Wi-Fi, GPS mobile navigation, camera,video camera, speech recognition, voice recorder, music player, nearfield communication and infrared blaster. Currently popular mobile OSsare Android, Symbian, Apple iOS, BlackBerry, MeeGo, Windows Phone, andBada. Mobile devices with mobile communications capabilities (e.g.smartphones) typically contain two mobile operating systems—a mainuser-facing software platform is supplemented by a second low-levelproprietary real-time operating system that operates the radio and otherhardware.

Android is an open source and Linux-based mobile operating system (OS)based on the Linux kernel that is currently offered by Google. With auser interface based on direct manipulation, Android is designedprimarily for touchscreen mobile devices such as smartphones and tabletcomputers, with specialized user interfaces for televisions (AndroidTV), cars (Android Auto), and wrist watches (Android Wear). The OS usestouch inputs that loosely correspond to real-world actions, such asswiping, tapping, pinching, and reverse pinching to manipulate on-screenobjects, and a virtual keyboard. Despite being primarily designed fortouchscreen input, it also has been used in game consoles, digitalcameras, and other electronics. The response to user input is designedto be immediate and provides a fluid touch interface, often using thevibration capabilities of the device to provide haptic feedback to theuser. Internal hardware such as accelerometers, gyroscopes and proximitysensors are used by some applications to respond to additional useractions, for example adjusting the screen from portrait to landscapedepending on how the device is oriented, or allowing the user to steer avehicle in a racing game by rotating the device by simulating control ofa steering wheel.

Android devices boot to the homescreen, the primary navigation andinformation point on the device, which is similar to the desktop foundon PCs. Android homescreens are typically made up of app icons andwidgets; app icons launch the associated app, whereas widgets displaylive, auto-updating content such as the weather forecast, the user'semail inbox, or a news ticker directly on the homescreen. A homescreenmay be made up of several pages that the user can swipe back and forthbetween, though Android's homescreen interface is heavily customizable,allowing the user to adjust the look and feel of the device to theirtastes. Third-party apps available on Google Play and other app storescan extensively re-theme the homescreen, and even mimic the look ofother operating systems, such as Windows Phone. The Android OS isdescribed in a publication entitled: “Android Tutorial”, downloaded fromtutorialspoint.com on July 2014, which is incorporated in its entiretyfor all purposes as if fully set forth herein.

iOS (previously iPhone OS) from Apple Inc. (headquartered in Cupertino,Calif., U.S.A.) is a mobile operating system distributed exclusively forApple hardware. The user interface of the iOS is based on the concept ofdirect manipulation, using multi-touch gestures. Interface controlelements consist of sliders, switches, and buttons. Interaction with theOS includes gestures such as swipe, tap, pinch, and reverse pinch, allof which have specific definitions within the context of the iOSoperating system and its multi-touch interface. Internal accelerometersare used by some applications to respond to shaking the device (onecommon result is the undo command) or rotating it in three dimensions(one common result is switching from portrait to landscape mode). TheiOS OS is described in a publication entitled: “IOS Tutorial”,downloaded from tutorialspoint.com on July 2014, which is incorporatedin its entirety for all purposes as if fully set forth herein.

RTOS. A Real-Time Operating System (RTOS) is an Operating System (OS)intended to serve real-time applications that process data as it comesin, typically without buffer delays. Processing time requirements(including any OS delay) are typically measured in tenths of seconds orshorter increments of time, and is a time bound system which has welldefined fixed time constraints. Processing is commonly to be done withinthe defined constraints, or the system will fail. They either are eventdriven or time sharing, where event driven systems switch between tasksbased on their priorities while time sharing systems switch the taskbased on clock interrupts. A key characteristic of an RTOS is the levelof its consistency concerning the amount of time it takes to accept andcomplete an application's task; the variability is jitter. A hardreal-time operating system has less jitter than a soft real-timeoperating system. The chief design goal is not high throughput, butrather a guarantee of a soft or hard performance category. An RTOS thatcan usually or generally meet a deadline is a soft real-time OS, but ifit can meet a deadline deterministically it is a hard real-time OS. AnRTOS has an advanced algorithm for scheduling, and includes a schedulerflexibility that enables a wider, computer-system orchestration ofprocess priorities. Key factors in a real-time OS are minimal interruptlatency and minimal thread switching latency; a real-time OS is valuedmore for how quickly or how predictably it can respond than for theamount of work it can perform in a given period of time.

Common designs of RTOS include event-driven, where tasks are switchedonly when an event of higher priority needs servicing; called preemptivepriority, or priority scheduling, and time-sharing, where task areswitched on a regular clocked interrupt, and on events; called roundrobin. Time sharing designs switch tasks more often than strictlyneeded, but give smoother multitasking, giving the illusion that aprocess or user has sole use of a machine. In typical designs, a taskhas three states: Running (executing on the CPU); Ready (ready to beexecuted); and Blocked (waiting for an event, I/O for example). Mosttasks are blocked or ready most of the time because generally only onetask can run at a time per CPU. The number of items in the ready queuecan vary greatly, depending on the number of tasks the system needs toperform and the type of scheduler that the system uses. On simplernon-preemptive but still multitasking systems, a task has to give up itstime on the CPU to other tasks, which can cause the ready queue to havea greater number of overall tasks in the ready to be executed state(resource starvation).

RTOS concepts and implementations are described in an Application NoteNo. RES05B00008-0100/Rec. 1.00 published January 2010 by RenesasTechnology Corp. entitled: “R8C FamilyGeneral RTOS Concepts”, in JAJATechnology Review article published February 2007 [1535-5535/$32.00] byThe Association for Laboratory Automation[doi:10.1016/j.jala.2006.10.016] entitled: “An Overview of Real-TimeOperating Systems”, and in Chapter 2 entitled: “Basic Concepts of RealTime Operating Systems” of a book published 2009[ISBN-978-1-4020-9435-4] by Springer Science+Business Media B. V.entitled: “Hardware-Dependent Software—Principles and Practice”, whichare all incorporated in their entirety for all purposes as if fully setforth herein.

QNX. One example of RTOS is QNX, which is a commercial Unix-likereal-time operating system, aimed primarily at the embedded systemsmarket. QNX was one of the first commercially successful microkerneloperating systems and is used in a variety of devices including cars andmobile phones. As a microkernel-based OS, QNX is based on the idea ofrunning most of the operating system kernel in the form of a number ofsmall tasks, known as Resource Managers. In the case of QNX, the use ofa microkernel allows users (developers) to turn off any functionalitythey do not require without having to change the OS itself; instead,those services will simply not run.

FreeRTOS. FreeRTOS™ is a free and open-source Real-Time Operating systemdeveloped by Real Time Engineers Ltd., designed to fit on small embeddedsystems and implements only a very minimalist set of functions: verybasic handle of tasks and memory management, and just sufficient APIconcerning synchronization. Its features include characteristics such aspreemptive tasks, support for multiple microcontroller architectures, asmall footprint (4.3 Kbytes on an ARM7 after compilation), written in C,and compiled with various C compilers. It also allows an unlimitednumber of tasks to run at the same time, and no limitation about theirpriorities as long as used hardware can afford it.

FreeRTOS™ provides methods for multiple threads or tasks, mutexes,semaphores and software timers. A tick-less mode is provided for lowpower applications, and thread priorities are supported. Four schemes ofmemory allocation are provided: allocate only; allocate and free with avery simple, fast, algorithm; a more complex but fast allocate and freealgorithm with memory coalescence; and C library allocate and free withsome mutual exclusion protection. While the emphasis is on compactnessand speed of execution, a command line interface and POSIX-like IOabstraction add-ons are supported. FreeRTOS™ implements multiple threadsby having the host program call a thread tick method at regular shortintervals.

The thread tick method switches tasks depending on priority and around-robin scheduling scheme. The usual interval is 1/1000 of a secondto 1/100 of a second, via an interrupt from a hardware timer, but thisinterval is often changed to suit a particular application. FreeRTOS™ isdescribed in a paper by Nicolas Melot (downloaded July 2015) entitled:“Study of an operating system: FreeRTOS—Operating systems for embeddeddevices”, in a paper (dated Sep. 23, 2013) by Dr. Richard Wall entitled:“Carebot PIC32 MX7ck implementation of Free RTOS”, FreeRTOS™ modules aredescribed in web pages entitled: “FreeRTOS™ Modules” published in thewww.freertos.org web-site dated 26 Nov. 2006, and FreeRTOS kernel isdescribed in a paper published 1 Apr. 2007 by Rich Goyette of CarletonUniversity as part of ‘SYSC5701: Operating System Methods for Real-TimeApplications’, entitled: “An Analysis and Description of the InnerWorkings of the FreeRTOS Kernel”, which are all incorporated in theirentirety for all purposes as if fully set forth herein.

SafeRTOS. SafeRTOS was constructed as a complementary offering toFreeRTOS, with common functionality but with a uniquely designedsafety-critical implementation. When the FreeRTOS functional model wassubjected to a full HAZOP, weakness with respect to user misuse andhardware failure within the functional model and API were identified andresolved. Both SafeRTOS and FreeRTOS share the same schedulingalgorithm, have similar APIs, and are otherwise very similar, but theywere developed with differing objectives. SafeRTOS was developed solelyin the C language to meet requirements for certification to IEC61508.SafeRTOS is known for its ability to reside solely in the on-chip readonly memory of a microcontroller for standards compliance. Whenimplemented in hardware memory, SafeRTOS code can only be utilized inits original configuration, so certification testing of systems usingthis OS need not re-test this portion of their designs during thefunctional safety certification process.

VxWorks. VxWorks is an RTOS developed as proprietary software anddesigned for use in embedded systems requiring real-time, deterministicperformance and, in many cases, safety and security certification, forindustries, such as aerospace and defense, medical devices, industrialequipment, robotics, energy, transportation, network infrastructure,automotive, and consumer electronics. VxWorks supports Intelarchitecture, POWER architecture, and ARM architectures. The VxWorks maybe used in multicore asymmetric multiprocessing (AMP), symmetricmultiprocessing (SMP), and mixed modes and multi-OS (via Type 1hypervisor) designs on 32- and 64-bit processors. VxWorks comes with thekernel, middleware, board support packages, Wind River Workbenchdevelopment suite and complementary third-party software and hardwaretechnologies. In its latest release, VxWorks 7, the RTOS has beenre-engineered for modularity and upgradeability so the OS kernel isseparate from middleware, applications and other packages. Scalability,security, safety, connectivity, and graphics have been improved toaddress Internet of Things (IoT) needs.

μC/OS. Micro-Controller Operating Systems (MicroC/OS, stylized as μC/OS)is a real-time operating system (RTOS) that is a priority-basedpreemptive real-time kernel for microprocessors, written mostly in theprogramming language C, and is intended for use in embedded systems.MicroC/OS allows defining several functions in C, each of which canexecute as an independent thread or task. Each task runs at a differentpriority, and runs as if it owns the central processing unit (CPU).Lower priority tasks can be preempted by higher priority tasks at anytime. Higher priority tasks use operating system (OS) services (such asa delay or event) to allow lower priority tasks to execute. OS servicesare provided for managing tasks and memory, communicating between tasks,and timing.

Vehicle. A vehicle is a mobile machine that transports people or cargo.Most often, vehicles are manufactured, such as wagons, bicycles, motorvehicles (motorcycles, cars, trucks, buses), railed vehicles (trains,trams), watercraft (ships, boats), aircraft and spacecraft. The vehiclemay be designed for use on land, in fluids, or be airborne, such asbicycle, car, automobile, motorcycle, train, ship, boat, submarine,airplane, scooter, bus, subway, train, or spacecraft. A vehicle mayconsist of, or may comprise, a bicycle, a car, a motorcycle, a train, aship, an aircraft, a boat, a spacecraft, a boat, a submarine, adirigible, an electric scooter, a subway, a train, a trolleybus, a tram,a sailboat, a yacht, or an airplane. Further, a vehicle may be abicycle, a car, a motorcycle, a train, a ship, an aircraft, a boat, aspacecraft, a boat, a submarine, a dirigible, an electric scooter, asubway, a train, a trolleybus, a tram, a sailboat, a yacht, or anairplane.

A vehicle may be a land vehicle typically moving on the ground, usingwheels, tracks, rails, or skies. The vehicle may be locomotion-basedwhere the vehicle is towed by another vehicle or an animal. Propellers(as well as screws, fans, nozzles, or rotors) are used to move on orthrough a fluid or air, such as in watercrafts and aircrafts. The systemdescribed herein may be used to control, monitor or otherwise be partof, or communicate with, the vehicle motion system. Similarly, thesystem described herein may be used to control, monitor or otherwise bepart of, or communicate with, the vehicle steering system. Commonly,wheeled vehicles steer by angling their front or rear (or both) wheels,while ships, boats, submarines, dirigibles, airplanes and other vehiclesmoving in or on fluid or air usually have a rudder for steering. Thevehicle may be an automobile, defined as a wheeled passenger vehiclethat carries its own motor, and primarily designed to run on roads, andhave seating for one to six people. Typically automobiles have fourwheels, and are constructed to principally transport of people.

Human power may be used as a source of energy for the vehicle, such asin non-motorized bicycles. Further, energy may be extracted from thesurrounding environment, such as solar powered car or aircraft, a streetcar, as well as by sailboats and land yachts using the wind energy.Alternatively or in addition, the vehicle may include energy storage,and the energy is converted to generate the vehicle motion. A commontype of energy source is a fuel, and external or internal combustionengines are used to burn the fuel (such as gasoline, diesel, or ethanol)and create a pressure that is converted to a motion. Another commonmedium for storing energy are batteries or fuel cells, which storechemical energy used to power an electric motor, such as in motorvehicles, electric bicycles, electric scooters, small boats, subways,trains, trolleybuses, and trams.

Aircraft. An aircraft is a machine that is able to fly by gainingsupport from the air. It counters the force of gravity by using eitherstatic lift or by using the dynamic lift of an airfoil, or in a fewcases, the downward thrust from jet engines. The human activity thatsurrounds aircraft is called aviation. Crewed aircraft are flown by anonboard pilot, but unmanned aerial vehicles may be remotely controlledor self-controlled by onboard computers. Aircraft may be classified bydifferent criteria, such as lift type, aircraft propulsion, usage andothers.

Aerostats are lighter than air aircrafts that use buoyancy to float inthe air in much the same way that ships float on the water. They arecharacterized by one or more large gasbags or canopies filled with arelatively low-density gas such as helium, hydrogen, or hot air, whichis less dense than the surrounding air. When the weight of this is addedto the weight of the aircraft structure, it adds up to the same weightas the air that the craft displaces. Heavier-than-air aircraft, such asairplanes, must find some way to push air or gas downwards, so that areaction occurs (by Newton's laws of motion) to push the aircraftupwards. This dynamic movement through the air is the origin of the termaerodyne. There are two ways to produce dynamic upthrust: aerodynamiclift and powered lift in the form of engine thrust.

Aerodynamic lift involving wings is the most common, with fixed-wingaircraft being kept in the air by the forward movement of wings, androtorcraft by spinning wing-shaped rotors sometimes called rotary wings.A wing is a flat, horizontal surface, usually shaped in cross-section asan aerofoil. To fly, air must flow over the wing and generate lift. Aflexible wing is a wing made of fabric or thin sheet material, oftenstretched over a rigid frame. A kite is tethered to the ground andrelies on the speed of the wind over its wings, which may be flexible orrigid, fixed, or rotary.

Gliders are heavier-than-air aircraft that do not employ propulsion onceairborne. Take-off may be by launching forward and downward from a highlocation, or by pulling into the air on a tow-line, either by aground-based winch or vehicle, or by a powered “tug” aircraft. For aglider to maintain its forward air speed and lift, it must descend inrelation to the air (but not necessarily in relation to the ground).Many gliders can ‘soar’—gain height from updrafts such as thermalcurrents. Common examples of gliders are sailplanes, hang gliders andparagliders. Powered aircraft have one or more onboard sources ofmechanical power, typically aircraft engines although rubber andmanpower have also been used. Most aircraft engines are eitherlightweight piston engines or gas turbines. Engine fuel is stored intanks, usually in the wings but larger aircraft also have additionalfuel tanks in the fuselage.

A propeller aircraft use one or more propellers (airscrews) to createthrust in a forward direction. The propeller is usually mounted in frontof the power source in tractor configuration but can be mounted behindin pusher configuration. Variations of propeller layout includecontra-rotating propellers and ducted fans. A Jet aircraft useairbreathing jet engines, which take in air, burn fuel with it in acombustion chamber, and accelerate the exhaust rearwards to providethrust. Turbojet and turbofan engines use a spinning turbine to driveone or more fans, which provide additional thrust. An afterburner may beused to inject extra fuel into the hot exhaust, especially on military“fast jets”. Use of a turbine is not absolutely necessary: other designsinclude the pulse jet and ramjet. These mechanically simple designscannot work when stationary, so the aircraft must be launched to flyingspeed by some other method. Some rotorcrafts, such as helicopters, havea powered rotary wing or rotor, where the rotor disc can be angledslightly forward so that a proportion of its lift is directed forwards.The rotor may, similar to a propeller, be powered by a variety ofmethods such as a piston engine or turbine. Experiments have also usedjet nozzles at the rotor blade tips.

A vehicle may include a hood (a.k.a. bonnet), which is the hinged coverover the engine of motor vehicles that allows access to the enginecompartment (or trunk on rear-engine and some mid-engine vehicles) formaintenance and repair. A vehicle may include a bumper, which is astructure attached, or integrated to, the front and rear of anautomobile to absorb impact in a minor collision, ideally minimizingrepair costs. Bumpers also have two safety functions: minimizing heightmismatches between vehicles and protecting pedestrians from injury. Avehicle may include a cowling, which is the covering of a vehicle'sengine, most often found on automobiles and aircraft. A vehicle mayinclude a dashboard (also called dash, instrument panel, or fascia),which is a control panel placed in front of the driver of an automobile,housing instrumentation and controls for operation of the vehicle. Avehicle may include a fender that frames a wheel well (the fenderunderside). Its primary purpose is to prevent sand, mud, rocks, liquids,and other road spray from being thrown into the air by the rotatingtire. Fenders are typically rigid and can be damaged by contact with theroad surface. Instead, flexible mud flaps are used close to the groundwhere contact may be possible. A vehicle may include a quarter panel(a.k.a. rear wing), which is the body panel (exterior surface) of anautomobile between a rear door (or only door on each side for two-doormodels) and the trunk (boot) and typically wraps around the wheel well.Quarter panels are typically made of sheet metal, but are sometimes madeof fiberglass, carbon fiber, or fiber-reinforced plastic. A vehicle mayinclude a rocker, which is the body section below the base of the dooropenings. A vehicle may include a spoiler, which is an automotiveaerodynamic device whose intended design function is to ‘spoil’unfavorable air movement across a body of a vehicle in motion, usuallydescribed as turbulence or drag. Spoilers on the front of a vehicle areoften called air dams. Spoilers are often fitted to race andhigh-performance sports cars, although they have become common onpassenger vehicles as well. Some spoilers are added to cars primarilyfor styling purposes and have either little aerodynamic benefit or evenmake the aerodynamics worse. The trunk (a.k.a. boot) of a car is thevehicle's main storage compartment. A vehicle door is a type of door,typically hinged, but sometimes attached by other mechanisms such astracks, in front of an opening, which is used for entering and exiting avehicle. A vehicle door can be opened to provide access to the opening,or closed to secure it. These doors can be opened manually, or poweredelectronically. Powered doors are usually found on minivans, high-endcars, or modified cars. Car glass includes windscreens, side and rearwindows, and glass panel roofs on a vehicle. Side windows can be eitherfixed or be raised and lowered by depressing a button (power window) orswitch or using a hand-turned crank.

The lighting system of a motor vehicle consists of lighting andsignaling devices mounted or integrated to the front, rear, sides, andin some cases, the top of a motor vehicle. This lights the roadway forthe driver and increases the conspicuity of the vehicle, allowing otherdrivers and pedestrians to see a vehicle's presence, position, size,direction of travel, and the driver's intentions regarding direction andspeed of travel. Emergency vehicles usually carry distinctive lightingequipment to warn drivers and indicate priority of movement in traffic.A headlamp is a lamp attached to the front of a vehicle to light theroad ahead. A chassis consists of an internal framework that supports amanmade object in its construction and use. An example of a chassis isthe underpart of a motor vehicle, consisting of the frame (on which thebody is mounted).

Autonomous car. An autonomous car (also known as a driverless car,self-driving car, or robotic car) is a vehicle that is capable ofsensing its environment and navigating without human input. Autonomouscars use a variety of techniques to detect their surroundings, such asradar, laser light, GPS, odometry, and computer vision. Advanced controlsystems interpret sensory information to identify appropriate navigationpaths, as well as obstacles and relevant signage. Autonomous cars havecontrol systems that are capable of analyzing sensory data todistinguish between different cars on the road, which is very useful inplanning a path to the desired destination. Among the potential benefitsof autonomous cars is a significant reduction in traffic collisions; theresulting injuries; and related costs, including a lower need forinsurance. Autonomous cars are also predicted to offer major increasesin traffic flow; enhanced mobility for children, the elderly, disabledand poor people; the relief of travelers from driving and navigationchores; lower fuel consumption; significantly reduced needs for parkingspace in cities; a reduction in crime; and the facilitation of differentbusiness models for mobility as a service, especially those involved inthe sharing economy.

Modern self-driving cars generally use Bayesian SimultaneousLocalization And Mapping (SLAM) algorithms, which fuse data frommultiple sensors and an off-line map into current location estimates andmap updates. SLAM with Detection and Tracking of other Moving Objects(DATMO), which also handles things such as cars and pedestrians, is avariant being developed by research at Google. Simpler systems may useroadside Real-Time Locating System (RTLS) beacon systems to aidlocalization. Typical sensors include LIDAR and stereo vision, GPS andIMU. Visual object recognition uses machine vision including neuralnetworks.

The term ‘Dynamic driving task’ includes the operational (steering,braking, accelerating, monitoring the vehicle and roadway) and tactical(responding to events, determining when to change lanes, turn, usesignals, etc.) aspects of the driving task, but not the strategic(determining destinations and waypoints) aspect of the driving task. Theterm ‘Driving mode’ refers to a type of driving scenario withcharacteristic dynamic driving task requirements (e.g., expresswaymerging, high speed, cruising, low speed traffic jam, closed-campusoperations, etc.). The term ‘Request to intervene’ refers tonotification by the automated driving system to a human driver that s/heshould promptly begin or resume performance of the dynamic driving task.

The SAE International standard J3016, entitled: “Taxonomy andDefinitions for Terms Related to On-Road Motor Vehicle Automated DrivingSystems” [Revised 2016-09], which is incorporated in its entirety forall purposes as if fully set forth herein, describes six differentlevels (ranging from none to fully automated systems), based on theamount of driver intervention and attentiveness required, rather thanthe vehicle capabilities. The levels are further described in a table 20a in FIG. 2a . Level 0 refers to automated system issues warnings buthas no vehicle control, while Level 1 (also referred to as “hands on”)refers to driver and automated system that shares control over thevehicle. An example would be Adaptive Cruise Control (ACC) where thedriver controls steering and the automated system controls speed. UsingParking Assistance, steering is automated while speed is manual. Thedriver must be ready to retake full control at any time. Lane KeepingAssistance (LKA) Type II is a further example of level 1 self-driving.

In Level 2 (also referred to as “hands off”), the automated system takesfull control of the vehicle (accelerating, braking, and steering). Thedriver must monitor the driving and be prepared to immediately interveneat any time if the automated system fails to respond properly. In Level3 (also referred to as“eyes off”), the driver can safely turn theirattention away from the driving tasks, e.g. the driver can text or watcha movie. The vehicle will handle situations that call for an immediateresponse, like emergency braking. The driver must still be prepared tointervene within some limited time, specified by the manufacturer, whencalled upon by the vehicle to do so. A key distinction is between level2, where the human driver performs part of the dynamic driving task, andlevel 3, where the automated driving system performs the entire dynamicdriving task. Level 4 (also referred to as “mind off”) is similar tolevel 3, but no driver attention is ever required for safety, i.e., thedriver may safely go to sleep or leave the driver's seat. Self-drivingis supported only in limited areas (geofenced) or under specialcircumstances, such as traffic jams. Outside of these areas orcircumstances, the vehicle must be able to safely abort the trip, i.e.,park the car, if the driver does not retake control. In Level 5 (alsoreferred to as “wheel optional”), no human intervention is required. Anexample would be a robotic taxi.

An autonomous vehicle and systems having an interface for payloads thatallows integration of various payloads with relative ease are disclosedin U.S. Patent Application Publication No. 2007/0198144 to Norris et al.entitled: “Networked multi-role robotic vehicle”, which is incorporatedin its entirety for all purposes as if fully set forth herein. There isa vehicle control system for controlling an autonomous vehicle,receiving data, and transmitting a control signal on at least onenetwork. A payload is adapted to detachably connect to the autonomousvehicle, the payload comprising a network interface configured toreceive the control signal from the vehicle control system over the atleast one network. The vehicle control system may encapsulate payloaddata and transmit the payload data over the at least one network,including Ethernet or CAN networks. The payload may be a laser scanner,a radio, a chemical detection system, or a Global Positioning Systemunit. In certain embodiments, the payload is a camera mast unit, wherethe camera communicates with the autonomous vehicle control system todetect and avoid obstacles. The camera mast unit may be interchangeable,and may include structures for receiving additional payload components.

Automotive electric and electronic technologies and systems aredescribed in a book published by Robert Bosch GmbH (5^(th) Edition, July2007) entitled: “Bosch Automotive Electric and Automotive Electronics”[ISBN-978-3-658-01783-5], which is incorporated in its entirety for allpurposes as if fully set forth herein.

ADAS. Advanced Driver Assistance Systems, or ADAS, are automotiveelectronic systems to help the driver in the driving process, such as toincrease car safety and more generally road safety using a safeHuman-Machine Interface. Advanced driver assistance systems (ADAS) aredeveloped to automate/adapt/enhance vehicle systems for safety andbetter driving. Safety features are designed to avoid collisions andaccidents by offering technologies that alert the driver to potentialproblems, or to avoid collisions by implementing safeguards and takingover control of the vehicle. Adaptive features may automate lighting,provide adaptive cruise control, automate braking, incorporateGPS/traffic warnings, connect to smartphones, alert driver to other carsor dangers, keep the driver in the correct lane, or show what is inblind spots.

There are many forms of ADAS available; some features are built intocars or are available as an add-on package. ADAS technology can be basedupon, or use, vision/camera systems, sensor technology, car datanetworks, Vehicle-to-vehicle (V2V), or Vehicle-to-Infrastructuresystems, and leverage wireless network connectivity to offer improvedvalue by using car-to-car and car-to-infrastructure data. ADAStechnologies or applications comprise: Adaptive Cruise Control (ACC),Adaptive High Beam, Glare-free high beam and pixel light, Adaptive lightcontrol such as swiveling curve lights, Automatic parking, Automotivenavigation system with typically GPS and TMC for providing up-to-datetraffic information, Automotive night vision, Automatic EmergencyBraking (AEB), Backup assist, Blind Spot Monitoring (BSM), Blind SpotWarning (BSW), Brake light or traffic signal recognition, Collisionavoidance system (such as Precrash system), Collision Imminent Braking(CIB), Cooperative Adaptive Cruise Control (CACC), Crosswindstabilization, Driver drowsiness detection, Driver Monitoring Systems(DMS), Do-Not-Pass Warning (DNPW), Electric vehicle warning sounds usedin hybrids and plug-in electric vehicles, Emergency driver assistant,Emergency Electronic Brake Light (EEBL), Forward Collision Warning(FCW), Heads-Up Display (HUD), Intersection assistant, Hill descentcontrol, Intelligent speed adaptation or Intelligent Speed Advice (ISA),Intelligent Speed Adaptation (ISA), Intersection Movement Assist (IMA),Lane Keeping Assist (LKA), Lane Departure Warning (LDW) (a.k.a. LineChange Warning—LCW), Lane change assistance, Left Turn Assist (LTA),Night Vision System (NVS), Parking Assistance (PA), Pedestrian DetectionSystem (PDS), Pedestrian protection system, Pedestrian Detection (PED),Road Sign Recognition (RSR), Surround View Cameras (SVC), Traffic signrecognition, Traffic jam assist, Turning assistant, Vehicularcommunication systems, Autonomous Emergency Braking (AEB), AdaptiveFront Lights (AFL), or Wrong-way driving warning.

ADAS is further described in Intel Corporation 2015 Technical WhitePaper (0115/MW/HBD/PDF 331817-001US) by Meiyuan Zhao of Security &Privacy Research, Intel Labs entitled: “Advanced Driver AssistantSystem—Threats, Requirements, Security Solutions”, and in a PhD Thesisby Alexandre Dugarry submitted on June 2004 to the Cranfield University,School of Engineering, Applied Mathematics and Computing Group,entitled: “Advanced Driver Assistance Systems—Information Management andPresentation”, which are both incorporated in their entirety for allpurposes as if fully set forth herein.

Wearables. As used herein, the term “wearable device” (or “wearable”)includes a body-borne device (or item) designed or intended to be wornby a human. Such devices are typically comfortably worn on, and arecarried or transported by, the human body, and are commonly used tocreate constant, convenient, seamless, portable, and mostly hands-freeaccess to electronics and computers. The wearable devices may be indirect contact with the human body (such as by touching, or attachingto, the body skin), or may be releasably attachable to clothes or otheritems intended or designed to be worn on the human body. In general, thegoal of wearable technologies is to smoothly incorporate functional,portable electronics and computers into individuals' daily lives.Wearable devices may be releasably attached to the human body usingattaching means such as straps, buckles, belts, or clasps. Alternativelyor in addition, wearable devices may be shaped, structured, or having aform factor to be body releasably mountable or attachable, such as usingeye-glass frames or headphones. Further, wearable devices may be wornunder, with, or on top of, clothing.

Wearable devices may interact as sensors or actuators with an organ orpart of the human body, such as a head mounted wearable device mayinclude a screen suspended in front of a user's eye, without providingany aid to the user's vision. Examples of wearable devices includewatches, glasses, contact lenses, pedometers, chest straps, wrist-bands,head bands, arm bands, belt, head wear, hats, glasses, watches,sneakers, clothing, pads, e-textiles and smart fabrics, headbands,beanies, and caps, as well as jewelry such as rings, bracelets, andhearing aid-like devices that are designed to look like earrings. Awearable device may be structured, designed, or have a form factor thatis identical to, substantially similar to, or is at least in partsubstitute to, a traditional wearable item.

A wearable device may be a headwear that may be structured, designed, orhave a form factor that is identical to, substantially similar to, or isat least in part substitute to, any headwear item. The headwear may beattached to, or be in contact with, a head part, such as a face, nose,right nostril, left nostril, right cheek, left cheek, right eye, lefteye, right ear, or left ear, nose, mouth, lip, forehead, or chin. Awearable device may be structured, designed, or have a form factor thatis identical to, substantially similar to, or is at least in partsubstitute to, a bonnet, a cap, a crown, a fillet, a hair cover, a hat,a helmet, a hood, a mask, a turban, a veil, or a wig.

A headwear device may be an eyewear that may be structured, designed, orhave a form factor that is identical to, substantially similar to, or isat least in part substitute to, any eyewear item, such as glasses,sunglasses, a contact lens, a blindfold, or a goggle. A headwear devicemay be an earpiece that may be structured, designed, or have a formfactor that is identical to, substantially similar to, or is at least inpart substitute to, any earpiece item, such as a hearing aid, aheadphone, a headset, or an earplug.

A wearable device may be releasably or permanently attach to, or be partof, a clothing article such as a tie, sweater, jacket, or hat. Theattachment may use taping, gluing, pinning, enclosing, encapsulating, orany other method of attachment or integration known in the art.Furthermore, in some embodiments, there may be an attachment elementsuch as a pin or a latch and hook system, of portion thereof (with thecomplementary element on the item to which it is to be affixed) or clip.In a non-limiting example, the attachment element has a clip-like designto allow attachment to pockets, belts, watches, bracelets, broaches,rings, shoes, hats, bike handles, necklaces, ties, spectacles, collars,socks, bags, purses, wallets, or cords.

A wearable device may be releasably or permanently attach to, or be partof, a top underwear such as a bra, camisole, or undershirt, a bottomunderwear such as a diaper, panties, plastic pants, slip, thong,underpants, boxer briefs, boxer shorts, or briefs, or a full-bodyunderwear such as bodysuit, long underwear, playsuit, or teddy.Similarly, a wearable device may be releasably or permanently attach to,or be part of, a headwear such as a Baseball cap, Beret, Cap, Fedora,hat, helmet, hood, knit cap, toque, turban, or veil. Similarly, awearable device may be releasably or permanently attach to, or be partof, a footwear such as an athletic shoe, boot, court shoe, dress shoe,flip-flops, hosiery, sandal, shoe, spats, slipper, sock, or stocking.Further, a wearable device may be releasably or permanently attach to,or be part of, an accessory such as a bandana, belt, bow tie, coinpurse, cufflink, cummerbund, gaiters, glasses, gloves, headband,handbag, handkerchief, jewellery, muff, necktie, pocket protector,pocketwatch, sash, scarf, sunglasses, suspenders, umbrella, wallet, orwristwatch.

A wearable device may be releasably or permanently attach to, or be partof, an outwear such as an apron, blazer, British warm, cagoule, cape,chesterfield, coat, covert coat, cut-off, duffle coat, flight jacket,gilet, goggle jacket, guards coat, Harrington jacket, hoodie, jacket,leather jacket, mess jacket, opera coat, overcoat, parka, paletot, peacoat, poncho, raincoat, robe, safari jacket, shawl, shrug, ski suit,sleeved blanket, smoking jacket, sport coat, trench coat, ulster coat,waistcoat, or windbreaker. Similarly, a wearable device may bereleasably or permanently attach to, or be part of, a suit (or uniform)such as an academic dress, ball dress, black tie, boilersuit, cleanroomsuit, clerical clothing, court dress, gymslip, jumpsuit, kasaya, labcoat, military uniform, morning dress, onesie, pantsuit, red sea rig,romper suit, school uniform, scrubs, stroller, tuxedo, or white tie.Further, a wearable device may be releasably or permanently attach to,or be part of, a dress such as a ball gown, bouffant gown, coatdress,cocktail dress, debutante dress, formal wear, frock, evening gown, gown,house dress, jumper, little black dress, princess line, sheath dress,shirtdress, slip dress, strapless dress, sundress, wedding dress, orwrap dress. Furthermore, a wearable device may be releasably orpermanently attach to, or be part of, a skirt such as an A-line skirt,ballerina skirt, denim skirt, men's skirts, miniskirt, pencil skirt,prairie skirt, rah-rah skirt, sarong, Skort, tutu, or wrap. In oneexample, a wearable device may be releasably or permanently attach to,or be part of, a trousers (or shorts) such as bell-bottoms, bermudashorts, bondage pants, capri pants, cargo pants, chaps, cycling shorts,dress pants, high water pants, lowrise pants, Jeans, jodhpurs, leggings,overall, Palazzo pants, parachute pants, pedal pushers, phat pants,shorts, slim-fit pants, sweatpants, windpants, or yoga pants. In oneexample, a wearable device may be releasably or permanently attach to,or be part of, a top such as a blouse, crop top, dress shirt, guayabera,guernsey, halterneck, henley shirt, hoodie, jersey, polo shirt, shirt,sleeveless shirt, sweater, sweater vest, t-shirt, tube top, turtleneck,or twinset.

A wearable device may be structured, designed, or have a form factorthat is identical to, substantially similar to, or is at least in partsubstitute to, a fashion accessory. These accessories may be purelydecorative, or have a utility beyond aesthetics. Examples of theseaccessories include, but are not limited to, rings, bracelets,necklaces, watches, watch bands, purses, wallets, earrings, body rings,headbands, glasses, belts, ties, tie bars, tie tacks, wallets, shoes,pendants, charms and bobbles. For example, wearable devices may also beincorporated into pockets, steering wheels, keyboards, pens, and bicyclehandles.

In one example, the wearable device may be shaped as, or integratedwith, a ring. The ring may comprise, consist essentially of or consistof a shank, which is the location that provides an opening for a finger,and a head, which comprises, consists essentially or consists ofornamental features of the ring and in some embodiments houses thesignaling assembly of the present device. The head may be of any shape,e.g., a regular sphere, truncated sphere, cube, rectangular prism,cylinder, triangular prism, cone, pyramid, barrel, truncated cone, domedcylinder, truncated cylinder, ellipsoid, regular polygon prism ortruncated three-dimensional polygon of e.g., 4-16 sides, such as atruncated pyramid (trapezoid), or combination thereof or it may be anirregular shape. Further, the head may comprise an upper face thatcontains and is configured to show one or more jewels and/or ornamentaldesigns.

A mobile communication device that comprises a fashion accessory and asignaling assembly is described in U.S. Patent Application PublicationNo. 2015/0349556 to Mercando et al. entitled: “Mobile CommunicationDevices”, which is incorporated in its entirety for all purposes as iffully set forth herein. The signaling assembly may be configured toprovide sensory stimuli such as a flashing LED light and a vibration.These stimuli may vary depending on the signal received from a remotecommunication device or from gestures made by a user or from informationstored in the mobile communication device.

A wearable fitness-monitoring device is described in U.S. Pat. No.8,948,832 to Hong et al. entitled: “Wearable Heart Rate Monitor”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The device including a motion sensor and a photoplethysmographic(PPG) sensor. The PPG sensor includes (i) a periodic light source, (ii)a photo detector, and (iii) circuitry determining a user's heart ratefrom an output of the photo detector. Some embodiments provide methodsfor operating a heart rate monitor of a wearable fitness-monitoringdevice to measure one or more characteristics of a heartbeat waveform.Some embodiments provide methods for operating the wearable fitnessmonitoring device in a low power state when the device determines thatthe device is not worn by a user. Some embodiments provide methods foroperating the wearable fitness-monitoring device in a normal power statewhen the device determines that the device is worn by a user.

In one example, a wearable device may use, or may be based on, aprocessor or a microcontroller that is designed for wearableapplications, such as the CC2650 SimpleLink™ Multistandard Wireless MCUavailable from Texas Instruments Incorporated (headquartered in Dallas,Tex., U.S.A.) and described in a Texas Instrument 2015 publication#SWRT022 entitled: “SimpleLink™ Ultra-Low Power—Wireless MicrocontrollerPlatform”, and in a Texas Instrument 2015 datasheet #SWRS158A (publishedFebruary 2015, Revised October 2015) entitled: “CC2650 SimpleLink™Multistandard Wireless MCU”, which are both incorporated in theirentirety for all purposes as if fully set forth herein.

Virtual Reality. Virtual Reality (VR) or virtual realities, also knownas immersive multimedia or computer-simulated reality, is a computertechnology that replicates an environment, real or imagined, andsimulates a user's physical presence and environment to allow for userinteraction. Virtual realities artificially create sensory experience,which can include sight, touch, hearing, and smell. Most up-to-datevirtual realities are displayed either on a computer monitor or with avirtual reality headset (also called head-mounted display), and somesimulations include additional sensory information and focus on realsound through speakers or headphones targeted towards VR users. Someadvanced haptic systems now include tactile information, generally knownas force feedback in medical, gaming and military applications.Furthermore, virtual reality covers remote communication environmentswhich provide virtual presence of users with the concepts oftelepresence and telexistence or a virtual artifact (VA) either throughthe use of standard input devices such as a keyboard and mouse, orthrough multimodal devices such as a wired glove or omnidirectionaltreadmills. The immersive environment can be similar to the real worldin order to create a lifelike experience—for example, in simulations forpilot or combat training—or it can differ significantly from reality,such as in VR games.

VR is described in an article published November 2009 in InternationalJournal of Automation and Computing 6(4), November 2009, 319-325 [DOI:10.1007/s11633-009-0319-9] by Ning-Ning Zhou and Yu-Long Deng entitled:“Virtual Reality: A State-of-the-Art Survey”, in a draft publicationauthored by Steven M. LaValle of the University of Illinois dated Jul.6, 2016 entitled: “VIRTUAL REALITY”, in an article by D. W. F. vanKrevelen and R. Poelman published 2010 in The International Journal ofVirtual Reality, 2010, 9(2):1-20 entitled: “A Survey of AugmentedReality—Technologies, Applications and Limitations”, in a paper by MosesOkechukwu Onyesolu and Felista Udoka Eze entitled: “UnderstandingVirtual Reality Technology: Advances and Applications” published 2011 bythe Federal University of Technology, Owerri, Imo State, Nigeria, in anarticle by Dr. Matthias Schmidt (Ed.) published in Advances andApplications, Advances in Computer Science and Engineering, [ISBN:978-953-307-173-2] by InTech, and in an Feb. 27, 2015 article by JamesWalker of Michigan Technological University entitled: “Everyday VirtualReality”, which are all incorporated in their entirety for all purposesas if fully set forth herein.

A method (50) of altering content provided to a user is described inU.S. Patent Application Publication No. 2007/0167689 to Ramadas et al.entitled: “Method and system for enhancing a user experience using auser's physiological state”, which is incorporated in its entirety forall purposes as if fully set forth herein. The method includes the stepsof creating (60) a user profile based on past physiological measurementsof the user, monitoring (74) at least one current physiologicalmeasurement of the user, and altering (82) the content provided to theuser based on the user profile and the at least one currentphysiological measurement. The user profile can be created by recordinga plurality of inferred or estimated emotional states (64) of the userwhich can include a time sequence of emotional states, stimulus contextsfor such states, and a temporal relationship between the emotional stateand the stimulus context. The content can be altered in response to theuser profile and measured physiological state by altering at least oneamong an audio volume, a video sequence, a sound effect, a video effect,a difficulty level, a sequence of media presentation.

A see-through, head mounted display and sensing devices cooperating withthe display detect audible and visual behaviors of a subject in a fieldof view of the device are described in U.S. Pat. No. 9,019,174 toJerauld entitled: “Wearable emotion detection and feedback system”,which is incorporated in its entirety for all purposes as if fully setforth herein. A processing device communicating with display and thesensors monitors audible and visual behaviors of the subject byreceiving data from the sensors. Emotional states are computed based onthe behaviors and feedback provided to the wearer indicating computedemotional states of the subject. During interactions, the device,recognizes emotional states in subjects by comparing detected sensorinput against a database of human/primate gestures/expressions, posture,and speech. Feedback is provided to the wearer after interpretation ofthe sensor input.

Method and devices for creating a sedentary virtual-reality system areprovided in U.S. Pat. No. 9,298,283 to Chau-Hsiung Lin, et al. entitled:“Sedentary virtual reality method and systems”, which is incorporated inits entirety for all purposes as if fully set forth herein. A userinterface is provided that allows for the intuitive navigation of thesedentary virtual-reality system based on the position of the usershead. The sedentary virtual-reality system can render a desktopcomputing environment. The user can switch the virtual-reality systeminto an augmented reality viewing mode or a real-world viewing mode thatallow the user to control and manipulate the rendered sedentaryenvironment. The modes can also change to allow the user greatersituational awareness and a longer duration of use.

HMD. A Head-Mounted Display (or Helmet-Mounted Display, for aviationapplications), both abbreviated HMD, is a display device, worn on thehead or as part of a helmet, that has a small display optic in front ofone (monocular HMD) or each eye (binocular HMD). There is also anOptical head-mounted display (OHMD), which is a wearable display thathas the capability of reflecting projected images as well as allowingthe user to see through it. A typical HMD has either one or two smalldisplays with lenses and semi-transparent mirrors embedded in a helmet,eyeglasses (also known as data glasses) or visor. The display units areminiaturized and may include CRT, LCDs, Liquid crystal on silicon(LCos), or OLED. Some vendors employ multiple micro-displays to increasetotal resolution and field of view.

HMDs differ in whether they can display just a Computer Generated Image(CGI), show live images from the real world or a combination of both.Most HMDs display only a computer-generated image, sometimes referred toas a virtual image. Some HMDs allow a CGI to be superimposed on areal-world view. This is sometimes referred to as augmented reality ormixed reality. Combining real-world view with CGI can be done byprojecting the CGI through a partially reflective mirror and viewing thereal world directly. This method is often called Optical See-Through.Combining real-world view with CGI can also be done electronically byaccepting video from a camera and mixing it electronically with CGI.This method is often called Video See-Through.

An optical head-mounted display uses an optical mixer, which is made ofpartly silvered mirrors. It has the capability of reflecting artificialimages as well as letting real images to cross the lens and let the userto look through it. Various techniques have existed for see-throughHMD's. Most of these techniques can be summarized into two mainfamilies: “Curved Mirror” based and “Waveguide” based. Various waveguidetechniques have existed for some time. These techniques includediffraction optics, holographic optics, polarized optics, and reflectiveoptics. Major HMD applications include military, governmental (fire,police, etc.) and civilian/commercial (medicine, video gaming, sports,etc.).

The Virtual Reality (VR) technology most fundamental to the proposedresearch is the Head-Mounted Display (HMD). An HMD is a helmet or visorworn by the user with two screens, one for each eye, so that astereoscopic “true 3D” image may be displayed to the user. This isachieved by displaying the same image in each screen, but offset by adistance equal to the distance between the user's eyes, mimicking howhuman vision perceives the world. HMDs can be opaque or see-through. Ina see-through HMD, the screens are transparent so that the user can seethe real world as well as what is being displayed on the screens.However, see-through HMDs often suffer from brightness problems thatmake them difficult to use in variable lighting conditions. Most opaqueHMD designs block out the real world so that the user can only see thescreens, thereby providing an immersive experience.

Some HMDs are used in conjunction with tracking systems. By tracking theuser's position or orientation (or both), the system can allow the userto move naturally via locomotion and by turning their head and body, andupdate the graphical display accordingly. This allows for naturalexploration of virtual environments without needing to rely on akeyboard, mouse, joystick, and similar interface hardware. Positionaltracking is often accomplished by attaching markers (such as infraredmarkers) to the HMD or the user's body and using multiple specialcameras to track the location of these markers in 3D space. Orientationtracking can be accomplished using an inertial tracker, which uses asensor to detect velocities on three axes. Some systems use acombination of optical and inertial tracking, and other trackingtechniques (e.g., magnetic) also exist. The output from the trackingsystems is fed into the computer rendering the graphical display so thatit can update the scene. Filtering is usually necessary to make the datausable since it comes in the form of noisy analog measurements. An HMD31 is pictorially depicted in FIG. 3b , and includes a horizontal strap34 a and a vertical strap 34 b for head wearing by a person. Awireless-capable HMD 31 a may include an antenna 33 a and an antenna 33b for wireless communication. The wireless-capable HMD 31 a is shownworn by a person 32 in a view 35 shown in FIG. 3 b.

Methods and systems for capturing an image are provided in U.S. PatentApplication Publication No. 2013/0222638 to Wheeler et al. entitled:“Image Capture Based on Gaze Detection”, which is incorporated in itsentirety for all purposes as if fully set forth herein. In one example,a head-mounted device (HMD) having an image capturing device, aviewfinder, a gaze acquisition system, and a controller may beconfigured to capture an image. The image capturing device may beconfigured to have an imaging field of view including at least a portionof a field of view provided by the viewfinder. The gaze acquisitionsystem may be configured to acquire a gaze direction of a wearer. Thecontroller may be configured to determine whether the acquired gazedirection is through the viewfinder and generate an image captureinstruction based on a determination that the acquired gaze directionindicates a gaze through the viewfinder. The controller may further beconfigured to cause the image capturing device to capture an image.

Methods and systems for capturing and storing an image are provided inU.S. Pat. No. 8,941,561 to Starner entitled: “Image Capture”, which isincorporated in its entirety for all purposes as if fully set forthherein. In one example, eye-movement data associated with aheadmountable device (HMD) may be received. The HMD may include animage-capture device arranged to capture image data corresponding to awearer-view associated with the HMD. In one case, the receivedeye-movement data may indicate sustained gaze. In this case, a locationof the sustained gaze may be determined, and an image including a viewof the location of the sustained gaze may be captured. At least oneindication of a context of the captured image, such as time and/orgeographic location of the HMD when the image was captured may bedetermined and stored in a data-item attribute database as part of arecord of the captured image. In a further example, movements associatedwith the HMD may also be determined and based on to determine sustainedgaze and the location of the sustained gaze.

A head mountable display (HMD) system is disclosed in U.S. PatentApplication Publication No. 2014/0362446 to Bickerstaff et al. entitled:“Electronic Correction Based on Eye Tracking”, which is incorporated inits entirety for all purposes as if fully set forth herein. The headmountable display (HMD) system comprises an eye position detectorcomprising one or more cameras configured to detect the position of eachof the HMD user's eyes; a dominant eye detector configured to detect adominant eye of the HMD user; and an image generator configured togenerate images for display by the HMD in dependence upon the HMD user'seye positions, the image generator being configured to apply a greaterweight to the detected position of the dominant eye than to the detectedposition of the non-dominant eye.

Methods and systems are described that involve a headmountable display(HMD) or an associated device determining the orientation of a person'shead relative to their body, are described in U.S. Pat. No. 9,268,136 toPatrick et al. entitled: “Use of Comparative Sensor Data to DetermineOrientation of Head Relative to Body”, which is incorporated in itsentirety for all purposes as if fully set forth herein. To do so,example methods and systems may compare sensor data from the HMD tocorresponding sensor data from a tracking device that is expected tomove in a manner that follows the wearer's body, such a mobile phonethat is located in the HMD wearer's pocket.

A Head Mountable Display (HMD) system in which images are generated fordisplay to the user is described in Patent Cooperation Treaty (PCT)International Application (IA) Publication No. WO 2014/199155 toAshforth et al. entitled: “Head-Mountable Apparatus and Systems”, whichis incorporated in its entirety for all purposes as if fully set forthherein. The head mountable display (HMD) system comprises a detectorconfigured to detect the eye position and/or orientation and/or the headorientation of the HMD wearer, and a controller configured to controlthe generation of images for display, at least in part, according to thedetection of the eye position and/or orientation and/or the headorientation of the HMD wearer; in which the controller is configured tochange the display of one or more image features according to whether ornot the user is currently looking at those image features, the imagefeatures are menu items or information items, by rendering an imagefeature so as to be more prominent on the display if the user is lookingat it, such that the image feature is enlarged, moved from a peripheraldisplay position, replaced by a larger image feature and/or broughtforward in a 3D display space if the user is looking at it.

An arrangement 40 of using a digital video camera 10 is shown in FIG. 4.The digital video camera 10, which may correspond to the camera 10 shownin FIG. 1, or may include part or all of the elements shown in thecamera 10 shown in FIG. 1, and is used to capture an image or a scene45, typically situated in front of the digital video camera 10. A videodata stream 43 that represents the captured image or scene 45 is outputby digital video camera 10. The digital video stream 43 may be furtherused or processed by a Combiner 41. In one example, the video datastream 43 is combined with another data stream 42 by the combiner 41, toproduce a combined output 44. For example, various information layersmay be added to the captured video stream 43, for producing the outputsignal 44.

However, due to various latencies induced in the digital video camera10, by any additional processing, or by communication of the signals,the digital data stream 43 received at the combiner 41 may be delayed bya time delay Δ′t, so if the actual digital video stream upon real-timecapturing with no delays is described as f 1(t), the inducement of thetime delay Δ′t results in a delayed digital video stream signalf1(t-Δ′t) at the input of the combiner 41. In one example, theadditional data 42 added to the captured digital data stream 43 may beassociated with timing (or timing related attributes such as position),such as real-time information or time-variant information. In such acase, the combiner 41 may improperly combine the incoming time-shiftedsignals, resulting non-accurate or wrong results.

The image processing may further include video enhancement such as videodenoising, image stabilization, unsharp masking, and super-resolution.Further, the image processing may include a Video Content Analysis(VCA), where the video content is analyzed to detect and determinetemporal events based on multiple images, and is commonly used forentertainment, healthcare, retail, automotive, transport, homeautomation, safety and security. The VCA functionalities include VideoMotion Detection (VMD), video tracking, and egomotion estimation, aswell as identification, behavior analysis, and other forms of situationawareness. A dynamic masking functionality involves blocking a part ofthe video signal based on the video signal itself, for example becauseof privacy concerns. The egomotion estimation functionality involves thedetermining of the location of a camera or estimating the camera motionrelative to a rigid scene, by analyzing its output signal. Motiondetection is used to determine the presence of a relevant motion in theobserved scene, while an object detection is used to determine thepresence of a type of object or entity, for example, a person or car, aswell as fire and smoke detection. Similarly, face recognition andAutomatic Number Plate Recognition may be used to recognize, andtherefore possibly identify persons or cars. Tamper detection is used todetermine whether the camera or the output signal is tampered with, andvideo tracking is used to determine the location of persons or objectsin the video signal, possibly with regard to an external reference grid.A pattern is defined as any form in an image having discerniblecharacteristics that provide a distinctive identity when contrasted withother forms. Pattern recognition may also be used, for ascertainingdifferences, as well as similarities, between patterns under observationand partitioning the patterns into appropriate categories based on theseperceived differences and similarities; and may include any procedurefor correctly identifying a discrete pattern, such as an alphanumericcharacter, as a member of a predefined pattern category. Further, thevideo or image processing may use, or be based on, the algorithms andtechniques disclosed in the book entitled: “Handbook of Image & VideoProcessing”, edited by Al Bovik, published by Academic Press, [ISBN:0-12-119790-5], and in the book published by Wiley-Interscience [ISBN:13-978-0-471-71998-4] (2005) by Tinku Acharya and Ajoy K. Ray entitled:“Image Processing—Principles and Applications”, which are bothincorporated in their entirety for all purposes as if fully set forthherein.

A system that can parse both telemetry data and corresponding encodedvideo data wherein the telemetry and video data are subsequentlysynchronized based upon temporal information, such as a time stamp, isdescribed in U.S. Patent Application Publication No. 2011/0090399 toWhitaker et al. entitled: “Data Search, Parser, and Synchronization ofVideo and Telemetry Data”, which is incorporated in its entirety for allpurposes as if fully set forth herein. The telemetry data and the videodata are originally unsynchronized and the data for each is acquired bya separate device. The acquiring devices may be located within orattached to an aerial vehicle. The system receives the telemetry datastream or file and the encoded video data stream or file and outputs aseries of synchronized video images with telemetry data. Thus, there istelemetry information associated with each video image. The telemetrydata may be acquired at a different rate than the video data. As aresult, telemetry data may be interpolated or extrapolated to createtelemetry data that corresponds to each video image. The present systemoperates in real-time, so that data acquired from aerial vehicles can bedisplayed on a map.

A system, apparatus, and method for combining video with telemetry datais described in international application published under the PatentCooperation Treaty (PCT) as WIPO PCT Publication No. WO 17214400 A1 toAGUILAR-GAMEZ et al. entitled: “Networked apparatus for real-time visualintegration of digital video with telemetry data feeds”, which isincorporated in its entirety for all purposes as if fully set forthherein. The video is received from a camera associated with a user at awireless device. Telemetry data associated with the video is received atthe wireless device. The telemetry data is time stamped as received. Thevideo is overlaid with the telemetry data to generate integrated videoutilizing the wireless device. The integrated video is communicated fromthe wireless device to one or more users.

A positional recording synchronization system is described in U.S.Patent Application Publication No. 2017/0301373 to Dat Tran et al.entitled: “Positional Recording Synchronization System”, which isincorporated in its entirety for all purposes as if fully set forthherein. The system can include: creating a time stamped telemetry pointfor an unmanned aerial vehicle; creating a time stamped recording;creating transformed data from the time stamped recording, thetransformed data being tiles for zooming or thumbnails; creating aflightpath array, an image metadata array, and a video metadata array;determining whether entries of the video metadata array match with theflightpath array; determining whether entries of the image metadataarray match with the flightpath array; synchronizing the time stampedtelemetry point with the time stamped recording based on either theentries of the image metadata array matching the flightpath array, theentries of the visualizer module matching the flightpath array, or acombination thereof; and displaying the time stamped telemetry point asa selection tool for calling, viewing, or manipulating the time stampedrecording on a display.

Condition detection using image processing may include receivingtelemetry data related to movement of a vehicle along a vehicle path isdescribed in U.S. Patent Application Publication No. 2018/0218214 toPESTUN et al. entitled: “Condition detection using image processing”,which is incorporated in its entirety for all purposes as if fully setforth herein. Condition detection using image processing may furtherinclude receiving images captured by the vehicle, and generating, basedon the telemetry data and the images, an altitude map for the images,and world coordinates alignment data for the images. Condition detectionusing image processing may further include detecting the entities in theimages, and locations of the entities detected in the images,consolidating the locations of the entities detected in the images todetermine a consolidated location for the entities detected in theimages, generating, based on the consolidated location, a mask relatedto the vehicle path and the entities detected in the images, andreconstructing three-dimensional entities model for certain types ofentities, based on the entities masks and world coordinates alignmentdata for the images.

A flight training image recording apparatus that includes a housingcomprising one or more cameras is described in U.S. Patent ApplicationPublication No. 2016/0027335 to Schoensee et al. entitled: “Flighttraining image recording apparatus”, which is incorporated in itsentirety for all purposes as if fully set forth herein. The housingand/or separate cameras in a cockpit are mounted in locations to captureimages of the pilot, the pilot's hands, the aircraft instrument paneland a field of view to the front of the aircraft. The recorded imagesare date and time synced along with aircraft location, speed and othertelemetry signals and cockpit and control tower audio signals into amultiplexed audio and visual stream. The multiplexed audio and videostream is downloaded either wirelessly to a remote processor or to aportable memory device which can be input to the remote processor. Theremote processor displays multiple camera images that are time-stampedsynced along with cockpit audio signals and aircraft telemetry for pilottraining.

An observation system that comprises at least one platform means and avideo or image sensor installed on said platform means is described ininternational application published under the Patent Cooperation Treaty(PCT) as WIPO PCT Publication No. WO 2007/135659 to Shechtman et al.entitled: “Clustering—based image registration”, which is incorporatedin its entirety for all purposes as if fully set forth herein. Thesystem is used in order to produce several images of an area of interestunder varying conditions and a computer system in order to performregistration between said images and wherein said system ischaracterized by a clustering-based image registration methodimplemented in said computer system, which includes steps of inputtingimages, detecting feature points, initial matching of feature pointsinto pairs, clustering feature point pairs, outlier rejection anddefining final correspondence of pairs of points.

Condition detection using image processing may include receiving a maskgenerated from images and telemetry data captured by a vehicle, analtitude map, and alignment data for the mask, is described in U.S.Patent Application Publication No. 2018/0260626 to PESTUN et al.entitled: “Condition detection using image processing”, which isincorporated in its entirety for all purposes as if fully set forthherein. The images may be related to movement of the vehicle along avehicle path and non-infrastructure entities along an infrastructureentity position of a corresponding infrastructure entity, and thetelemetry data may include movement log information related to themovement of the vehicle along the vehicle path. Condition detectionusing image processing may further include using the mask related to thevehicle path and the non-infrastructure entities, and an infrastructurerule to detect a risk related to the infrastructure entity by analyzingthe mask related to the vehicle path and the non-infrastructureentities, and the infrastructure rule, and determining whether theinfrastructure rule is violated.

An Ethernet-compatible synchronization process between isolated digitaldata streams assures synchronization by embedding an available time codefrom a first stream into data locations in a second stream that areknown a priori to be unneeded, is described in U.S. Patent ApplicationPublication No. 2010/0067553 to McKinney et al. entitled:“Synchronization of video with telemetry signals method and apparatus”,which is incorporated in its entirety for all purposes as if fully setforth herein. Successive bits of time code values, generated as a stepin acquiring and digitizing analog sensor data, are inserted intoleast-significant-bit locations in a digitized audio stream generatedalong with digitized image data by a digital video process. Theoverwritten LSB locations are shown to have no discernable effect onaudio reconstructed from the Ethernet packets. Telemetry recovery is thereverse of the embedment process, and the data streams are readilysynchronized by numerical methods.

A method for producing images is described in U.S. Patent ApplicationPublication No. 2007/0285438 to Kanowitz entitled: “Frame grabber”,which is incorporated in its entirety for all purposes as if fully setforth herein. The method involves acquiring images, acquiring datacorresponding to the location of the acquired images, and transferringthe images and data to a frame grabber. The method also involvescombining the images and data within the frame grabber to provide aplurality of imagery products.

An optical device is described in U.S. Patent Application PublicationNo. 2004/0155993 to Cueff et al. entitled: “Optical device, particularlya liquid-crystal imaging device, and mirror therefor”, which isincorporated in its entirety for all purposes as if fully set forthherein. The invention described relates to the field of optical devices,in particular liquid crystal imagers, as well as the mirrors associatedwith these optical devices. The optical device is angled, and includesat least one lamp (3) and a channel (9) guiding at least some of thelight coming from the lamp (3), as well as a mirror (12) in an angledpart of the optical device, consisting of a sheet which is folded sothat, on the one hand, it can be partially introduced into the channel(9), and, on the other hand, once introduced into the channel (9) andimmobilized therein, it can reflect some of the light coming from thelamp (3) into a determined direction. The invention may, in particular,be applied to liquid crystal imagers for military aircraft.

Systems and methods for analyzing a game application are disclosed inU.S. Patent Application Publication No. 2017/0266568 to Lucas et al.entitled: “Synchronized video with in game telemetry”, which isincorporated in its entirety for all purposes as if fully set forthherein. While the game application is executed in a gameplay session,embodiment of the systems and methods can acquire data associated withthe game application. The data acquired during the gameplay session maybe associated with a session identifier. Different types of data (suchas telemetry data and video data) can be linked together using thetimestamps of the gameplay session. A user can choose a timestamp of thegameplay session to view the data associated with that timestamp. Incertain embodiments, the systems and methods can associate an event withone or more timestamps. When a user chooses the event, the systems andmethods can automatically display event data starting from the beginningof the event.

A video recording method capable of synchronously merging information ofa barometer and positioning information into a video in real time isdisclosed in Chinese Patent Application Publication No. CN105163056Aentitled: “Video recording method capable of synchronously merginginformation of barometer and positioning information into video in realtime”, which is incorporated in its entirety for all purposes as iffully set forth herein. According to the method, video information,audio information, and air pressure information, altitude information,grid location coordinate information and speed information of a motioncamera in real time are acquired, coding processing on the videoinformation is carried out to output a first video flow, codingprocessing on the audio information is carried out to output an audioflow synchronization with the first video flow, coding processing on theair pressure information, the altitude information, the grid locationcoordinate information and the speed information is carried out tooutput an air pressure altitude data flow synchronization with the firstvideo flow and a coordinate speed data flow, through synthesis, a secondvideo flow containing synchronization air pressure, altitude, gridlocation coordinate and speed information is outputted, and an audio andvideo file containing the second video flow and the audio flow arefinally outputted. Through the method, the air pressure information, thealtitude information, the grid location coordinate information and thespeed information of the motion camera are merged in real time into thevideo through synchronization coding, so subsequent edition, managementand analysis on the video are conveniently carried out.

Electronic circuits and components are described in a book by Wikipediaentitled: “Electronics” downloaded from en.wikibooks.org dated Mar. 15,2015, which is incorporated in its entirety for all purposes as if fullyset forth herein.

Each of the methods or steps herein, may consist of, include, be partof, be integrated with, or be based on, a part of, or the whole of, thesteps, functionalities, or structure (such as software) described in thepublications that are incorporated in their entirety herein. Further,each of the components, devices, or elements herein may consist of,integrated with, include, be part of, or be based on, a part of, or thewhole of, the components, systems, devices or elements described in thepublications that are incorporated in their entirety herein.

In consideration of the foregoing, it would be an advancement in the artto provide methods and systems for estimating the latency or delay,either absolutely or relatively, or to use the estimation forsynchronization of a delayed digital video stream for low-error timealignment, that are simple, intuitive, small, secure, cost-effective,reliable, provide lower power consumption, provide lower CPU and/ormemory usage, easy to use, reduce latency, faster, has a minimum partcount, minimum hardware, and/or uses existing and available components,protocols, programs and applications for providing better quality ofservice, better or optimal resources allocation, and provides a betteruser experience.

SUMMARY

A method for estimating a delay of a video data stream from a DigitalVideo Camera (DVC) may be used with a physical phenomenon that mayaffect the video camera or the scene captured by the video camera. Anymethod herein may comprise receiving, from the digital video camera, thevideo data stream; producing, by a video processor, a first signal thatestimates the physical phenomenon value, by processing the video datastream for detecting the effect of the physical phenomenon on thecaptured video; receiving, from a sensor, a second signal that isresponsive to the physical phenomenon value; estimating a positive ornegative time delay value between the first and second signals bycomparing therebetween; and combining the video data stream withadditional data by synchronizing using the estimated time delay value.

A non-transitory tangible computer readable storage media may comprise acode to perform part of, or whole of, the steps of any method herein.Alternatively or in addition, any device herein may be housed in asingle enclosure and may comprise the digital camera, a memory forstoring computer-executable instructions, and a processor for executingthe instructions, and the processor may be configured by the memory toperform acts comprising part of, or whole of, any method herein. Anyapparatus, device, or enclosure herein may be a portable or a hand-heldenclosure, and the may be battery-operated, such as a notebook, a laptopcomputer, a media player, a cellular phone, a Personal Digital Assistant(PDA), or an image processing device. Any method herein may be used witha memory or a non-transitory tangible computer readable storage mediafor storing computer executable instructions that may comprise at leastpart of the method, and a processor for executing part of, or all of,the instructions. Any non-transitory computer readable medium may behaving computer executable instructions stored thereon, and theinstructions may include the steps of any method herein.

Any time delay value herein may be estimated in response to an event,and the time delay value may be continuously used for any combining orother manipulation herein. Any time delay value herein may be estimatedin response to a user control, in response to a power-up process, or maybe continuously estimated and used for any combining or manipulating.Any time delay value herein may be periodically estimated, such as everyat least 1 second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30seconds, 1 minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30minutes, 1 hour, 2 hours, 5 hours, 10 hours, 1 day, 2 days, 4 days, 1week, 2 weeks, 3 weeks, or 1 months, or every no more than 1 second, 2seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2,minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1 hour, 2 hours,5 hours, 10 hours, 1 day, 2 days, 4 days, 1 week, 2 weeks, 3 weeks, or 1months.

Any comparing herein may comprise comparing of the first and secondsignals during a time interval, and the time-interval may be less than 1millisecond, 2 milliseconds, 5 milliseconds, 10 milliseconds, 20milliseconds, 30 milliseconds, 50 milliseconds, 100 milliseconds, 200milliseconds, 500 milliseconds, 1 second, 2 seconds, 5 seconds, 10seconds, 20 seconds, 30 seconds, 1 minute, 2, minutes, 5 minutes, 10minutes, 20 minutes, 30 minutes, 1 hour, 2 hours, 5 hours, or 10 hours,or the time-interval may be more than 1 millisecond, 2 milliseconds, 5milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 50milliseconds, 100 milliseconds, 200 milliseconds, 500 milliseconds, 1second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1hour, 2 hours, 5 hours, or 10 hours.

Any comparing herein may comprise, may be based on, or may use,convolution, correlation, or a cross-correlation operation on the firstand second signals, and any comparing herein may comprise, may consistof, may be based on, or may use, calculating or estimating across-correlation coefficient of the first and second signals. Anyestimating herein of any time delay value may comprises, may consist of,may be based on, or may use, selecting a time delay value that resultsin a maximum value of the cross-correlation coefficient value when timeshifting the first or second signal by the time delay value. Further,any comparing herein may comprise, may consist of, may be based on, ormay use, Cross-Correlation (CC), Phase Transform (PHAT), MaximumLikelihood estimator (ML), Adaptive Least Mean Square filter (LMS), orAverage Square Difference Function (ASDF).

Any method herein may be used with a third signal that may be based on,may be a function of, may be in response, may be equal to, or may be thesame as, the first signal. Alternatively or in addition, any methodherein may be used with a fourth signal that may be based on, may be afunction of, may be in response, may be equal to, or may be the same as,the second signal. Any comparing herein may comprise detecting oridentifying a first event in the third signal at a first time point anddetecting or identifying a second event in the fourth signal at a secondtime point. Any time delay value herein may be estimated based on, equalto, or a function of, of a time different between the first and secondtime points. The first event or the second event may comprise detectingor identifying a peak value in the respective third or fourth signal.Alternatively or in addition, any method herein may be used with athreshold value, and the first event or the second event may comprisedetecting or identifying a threshold value crossing in the respectivethird or fourth signal. The third signal may be the first signal and thefourth signal may be the second signal. Any method herein may furthercomprise producing the third signal by applying a time-domain analysisor manipulation to the first signal, and producing the fourth signal byapplying a time-domain analysis or manipulation to the second signal.

Any time-domain analysis or manipulation herein may comprise detectingzero crossings, peak amplitude, rise-time, energy, a Mel-FrequencyAnalysis, calculating Mel-Frequency Cepstral Coefficients (MFCC), usinga Linear Predictive Coding (LPC), calculating LPC coefficients, adiscrete, continuous, monotonic, non-monotonic, elementary, algebraic,linear, polynomial, quadratic, Cubic, Nth-root based, exponential,transcendental, quintic, quartic, logarithmic, hyperbolic, ortrigonometric function. Any method herein may further comprise producingthe third signal by applying a frequency-domain analysis or manipulationto the first signal and producing the fourth signal by applying afrequency-domain analysis or manipulation to the second signal, and anyfrequency-domain analysis or manipulation herein may comprise Fourierseries, Fourier transform, Laplace transform, Z transform, or Wavelettransform.

Any digital video camera herein may comprise an optical lens forfocusing received light, the lens being mechanically oriented to guide acaptured image; a photosensitive image sensor array disposedapproximately at an image focal point plane of the optical lens forcapturing the image and producing an analog signal representing theimage; and an analog-to-digital (A/D) converter coupled to the imagesensor array for converting the analog signal to the video data stream.Any camera or image sensor array herein may be operative to respond to avisible or non-visible light, and any invisible light herein may beinfrared, ultraviolet, X-rays, or gamma rays. Any image sensor arrayherein may comprise, may use, or may be based on, semiconductor elementsthat use the photoelectric or photovoltaic effect, such asCharge-Coupled Devices (CCD) or Complementary Metal-Oxide-SemiconductorDevices (CMOS) elements. Any video camera herein may consist of, maycomprise, or may be based on, a Light Detection And Ranging (LIDAR)camera or scanner, or a thermal camera.

Any digital video camera herein may further comprise an image processorcoupled to the image sensor array for providing the video data streamaccording to a digital video format, which may use, may be compatiblewith, may be according to, or may be based on, TIFF (Tagged Image FileFormat), RAW format, AVI, DV, MOV, WMV, MP4, DCF (Design Rule for CameraFormat), ITU-T H.261, ITU-T H.263, ITU-T H.264, ITU-T CCIR 601, ASF,Exif (Exchangeable Image File Format), or DPOF (Digital Print OrderFormat) standard. Further, any video data stream herein may be in aHigh-Definition (HD) or Standard-Definition (SD) format. Alternativelyor in addition, any video data stream herein may be based on, may becompatible with, or may be according to, ISO/IEC 14496 standard, MPEG-4standard, or ITU-T H.264 standard.

Any method herein may be used with a video compressor coupled to thedigital video camera for compressing the video data stream, and anyvideo compressor herein may perform a compression scheme that may uses,or may be based on, intraframe or interframe compression, and whereinthe compression is lossy or non-lossy. Further, any compression schemeherein may use, may be compatible with, or may be based on, at least onestandard compression algorithm which is selected from a group consistingof: JPEG (Joint Photographic Experts Group) and MPEG (Moving PictureExperts Group), ITU-T H.261, ITU-T H.263, ITU-T H.264 and ITU-T CCIR601.

Any producing of the first signal herein may comprise, may be based on,or may use, a Video Content Analysis (VCA) algorithm, which may includeVideo Motion Detection (VMD), video tracking, egomotion estimation,identification, behavior analysis, situation awareness, dynamic masking,motion detection, object detection, face recognition, automatic numberplate recognition, tamper detection, video tracking, or patternrecognition.

Any sensor herein may respond to an object that may be a gas, an air, aliquid, or a solid, and may consist of, may be based on, or maycomprise, a nanosensor, a crystal, a Micro Electro-Mechanical Systems(MEMS), or a semiconductor. Further, any sensor herein may respond to atime-dependent characteristic of the phenomenon that may be atime-integrated, an average, an RMS (Root Mean Square) value, afrequency, a period, a duty-cycle, a time-integrated, or atime-derivative of the phenomenon value. Alternatively or in addition,any sensor herein may respond to a space-dependent characteristic of thephenomenon that may be a pattern, a linear density, a surface density, avolume density, a flux density, a current, a direction, a rate of changein a direction, or a flow. Further, any sensor herein may be apiezoelectric sensor that may include single crystal material or apiezoelectric ceramics and uses a transverse, longitudinal, or sheareffect mode of the piezoelectric effect. Any method herein may be usedwith multiple sensors that may be arranged as a directional sensor arrayand may be operative to estimate the number, magnitude, frequency,Direction-Of-Arrival (DOA), distance, or speed of the phenomenonimpinging the sensor array.

Any sensor herein and any video camera herein may be mechanicallycoupled so that they are jointly affected by a movement or motion.Further, any sensor herein may be directly mechanically attached to anyvideo camera herein, or may be included in an enclosure of any videocamera herein. Alternatively or in addition, the sensor and the videocamera may be mechanically attached to a structure or frame, such as toa structure or frame of a single enclosure housing the sensor and thevideo camera. Any physical phenomenon herein may comprise a positioning,orientation, movement, or motion of the video camera, such as anabsolute or relative orientation or angular velocity of the videocamera. Any physical phenomenon herein may comprise an absolute orrelative yaw, pitch, roll, yaw rate, pitch rate, or roll rate, oralternatively absolute or relative location, position, speed oracceleration along a vertical, transverse, or longitudinal axis.Alternatively or in addition, any physical phenomenon herein maycomprise absolute or relative pan, tilt, or zoom of the video camera.Alternatively or in addition, any physical phenomenon herein maycomprise periodic or random mechanical vibration that may be free,forced, or damped vibration.

Any physical phenomenon herein may comprise a positioning, orientation,movement, or motion of the video camera, and any sensor herein may bemechanically attached to the video camera for sensing the positioning,orientation, movement, or motion of the video camera. Any sensor hereinmay use, may comprise, may consist of, or may be based on, a clinometerthat may use, may comprise, may consist of, or may be based on, anaccelerometer, a pendulum, or a gas bubble in liquid. Any sensor hereinmay use, may comprise, may consist of, or may be based on, an angularrate sensor, and any sensor herein may use, may comprise, may consistof, or may be based on, piezoelectric, piezoresistive, capacitive, MEMS,or electromechanical sensor. Alternatively or in addition, any sensorherein may use, may comprise, may consist of, or may be based on, aninertial sensor that may use, may comprise, may consist of, or may bebased on, one or more accelerometers, one or more gyroscopes, one ormore magnetometers, or an Inertial Measurement Unit (IMU).

Any sensor herein may use, may comprise, may consist of, or may be basedon, a single-axis, 2-axis or 3-axis accelerometer, which may use, maycomprise, may consist of, or may be based on, a piezoresistive,capacitive, Micro-mechanical Electrical Systems (MEMS), orelectromechanical accelerometer. Any accelerometer herein may beoperative to sense or measure the video camera mechanical orientation,vibration, shock, or falling, and may comprise, may consist of, may use,or may be based on, a piezoelectric accelerometer that utilizes apiezoelectric effect and comprises, consists of, uses, or is based on,piezoceramics or a single crystal or quartz. Alternatively or inaddition, any sensor herein may use, may comprise, may consist of, ormay be based on, a gyroscope that may use, may comprise, may consist of,or may be based on, a conventional mechanical gyroscope, a Ring LaserGyroscope (RLG), or a piezoelectric gyroscope, a laser-based gyroscope,a Fiber Optic Gyroscope (FOG), or a Vibrating Structure Gyroscope (VSG).

Any physical phenomenon herein may comprise an environmental condition,and any video camera herein and any sensor herein may be closely locatedto be jointly affected by the environmental condition, and a distancebetween the video camera and the sensor may be less than 1 cm(centimeter), 2 cm, 3 cm, 5 cm, 8 cm, 10 cm, 20 cm, 30 cm, 50 cm, 80 cm,1 m (meter), 2 m, 3 m, 5 m, 8 m, 10 m, 20 m, 30 m, 50 m, 80 m, 100 m,200 m, 300 m, 500 m, 800 m, or 1 Km (kilometer). Alternatively or inaddition, any sensor herein may be mechanically attached to any videocamera herein, or may be included in the video camera enclosure, such aswhere the sensor and the video camera, are housed within a singleenclosure. Any physical phenomenon herein may comprise, or may consistof, a temperature, and any sensor herein may comprise, may consist of,may use, or may be based on, a thermoelectric sensor that may respond toa temperature or to a temperature gradient of an object usingconduction, convection, or radiation. Any thermoelectric sensor hereinmay comprise, may consist of, may use, or may be based on, a PositiveTemperature Coefficient (PTC) thermistor, a Negative TemperatureCoefficient (NTC) thermistor, a thermocouple, a quartz crystal, or aResistance Temperature Detector (RTD).

Any environmental condition herein may comprise, or may consist of, aradiation, and any sensor herein may comprise, may consist of, may use,or may be based on, a radiation sensor that may respond toradioactivity, nuclear radiation, alpha particles, beta particles, orgamma rays, and may comprise, may consist of, may use, or may be basedon, gas ionization. Alternatively or in addition, any environmentalcondition herein may comprise, or may consist of, an ambient light, andany sensor herein may comprise, may consist of, may use, or may be basedon, a photoelectric sensor that may respond to a visible or an invisiblelight, and the invisible light may be infrared, ultraviolet, X-rays, orgamma rays. Further, any photoelectric sensor herein may comprise, mayconsist of, may use, or may be based on, a photoelectric or photovoltaiceffect, and may consist of, or may comprise, a semiconductor componentthat may consist of, or may comprise, a photodiode, a phototransistor,or a solar cell. Further, any photoelectric sensor herein may comprise,may consist of, may use, or may be based on, Charge-Coupled Device (CCD)or a Complementary Metal-Oxide Semiconductor (CMOS) element.Alternatively or in addition, any environmental condition herein maycomprise, or may consist of, humidity, and any sensor herein maycomprise, may consist of, may use, or may be based on, a humiditysensor. Further, any humidity sensor herein may comprise, may consistof, may use, or may be based on, a hygrometer or a humidistat, and mayrespond to an absolute, relative, or specific humidity. Alternatively orin addition, any environmental condition herein may comprise, or mayconsist of, an atmospheric condition, and any sensor herein maycomprise, may consist of, may use, or may be based on, an atmosphericsensor.

Any physical phenomenon herein may affect the scene captured by thevideo camera, such as a motion or a position of an element in the scenethat may be captured by the video camera. Any producing herein of thefirst signal may comprise, may be based on, or may use, a Video ContentAnalysis (VCA) algorithm that may comprise detecting or estimating themotion or the position of the element in the scene. Further, thephysical phenomenon may comprise a wind, and any sensor herein may beoperative to sense, detect, or measure the wind speed or direction.

Any digital camera herein and any sensor herein may be housed in a firstsingle enclosure. Any video processor herein may be housed in the firstsingle enclosure, the producing of the first signal may be performed inthe first single enclosure, the estimating of the time delay value maybe performed in the first single enclosure, or the combining of thevideo data stream with the additional data may be performed in the firstsingle enclosure. Alternatively or in addition, any combining herein ofthe video data stream with the additional data may be performed in asecond single enclosure that is distinct and separate from the firstsingle enclosure, and any estimating herein of the time delay value maybe performed in the second single enclosure. Further, any videoprocessor herein may be housed in the second single enclosure, or anyproducing herein of the first signal may be performed in the secondsingle enclosure.

Any method herein may be used with a device having a single enclosure,and the single enclosure may comprise any digital camera herein and anysensor herein. Any video processor herein may be part of the device, theproducing herein of the first signal may be performed in the device, theestimating herein of the time delay value may be performed in thedevice, or any combining herein of the video data stream with theadditional data may be performed in the device.

Any single enclosure herein may be a hand-held enclosure or a portableenclosure, or may be a surface mountable enclosure. Further, any deviceor enclosure herein may consist or, may comprise, or may be part of, atleast one of a wireless device, a notebook computer, a laptop computer,a media player, a Digital Still Camera (DSC), a Digital video Camera(DVC or digital camcorder), a Personal Digital Assistant (PDA), acellular telephone, a digital camera, a video recorder, or a smartphone.Furthermore, any device or enclosure herein may consist or, maycomprise, or may be part of, a smartphone that comprises, or is basedon, an Apple iPhone 6 or a Samsung Galaxy S6. Any method herein maycomprise operating of an operating system that may be a mobile operatingsystem, such as Android version 2.2 (Froyo), Android version 2.3(Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version4.2 (Jelly Bean), Android version 4.4 (KitKat)), Apple iOS version 3,Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple iOSversion 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phoneversion 8, Microsoft Windows® Phone version 9, or Blackberry® operatingsystem. Alternatively or in addition, any operating system may be aReal-Time Operating System (RTOS), such as FreeRTOS, SafeRTOS, QNX,VxWorks, or Micro-Controller Operating Systems (μC/OS).

Any device herein may be wearable on a person, such as on an organ ofthe person head, and the organ may be an eye, ear, face, cheek, nose,mouth, lip, forehead, or chin. Any device herein may include anenclosure that may be constructed to have a form substantially similarto, may be constructed to have a shape allowing mounting or wearingidentical or similar to, or may be constructed to have a form to atleast in part substitute for, headwear, eyewear, or earpiece.

Any headwear herein may consist of, may be structured as, or maycomprise, a bonnet, a cap, a crown, a fillet, a hair cover, a hat, ahelmet, a hood, a mask, a turban, a veil, or a wig. Any eyewear hereinmay consist of, may be structured as, or may comprise, glasses,sunglasses, a contact lens, a blindfold, or a goggle. Any earpieceherein may consist of, may be structured as, or may comprise, a hearingaid, a headphone, a headset, or an earplug. Any enclosure herein may bepermanently or releseably attachable to, or is part of, a clothing pieceof a person, and any attaching herein may use taping, gluing, pinning,enclosing, encapsulating, a pin, or a latch and hook clip. Any clothingpiece herein may be a top, bottom, or full-body underwear, or aheadwear, a footwear, an accessory, an outwear, a suit, a dress, askirt, or a top. Any device herein may comprise an annular memberdefining an aperture therethrough that is sized for receipt therein of apart of a human body. Any human body part herein may be part of a humanhand that may consist of, or may comprise, an upper arm, elbow, forearm,wrist, or a finger. Alternatively or in addition, any human body partherein may be part of a human head or neck that may consist of, or maycomprise, a forehead, ear, skull, or face. Alternatively or in addition,any human body part herein may be a part of a human thorax or abdomenthat may consist of, or may comprise, a waist or hip. Alternatively orin addition, any human body part herein may be part of a human leg orfoot that may consist of, or may comprise, a thigh, calf, ankle, instep,knee, or toe.

Any digital camera herein and any sensor herein may be part of, may bemounted in, or may be attached to, a vehicle. Further, any videoprocessor herein may be part of the vehicle, any producing herein of thefirst signal may be performed in the vehicle, any estimating herein ofthe time delay value may be performed in the vehicle, or any combiningherein of the video data stream with the additional data may beperformed in the vehicle.

Any method herein may be used with an automotive navigation system in avehicle, and any device herein may be part of, or may comprise,automotive navigation system and is installable, or mountable, in avehicle, and any user herein may be a driver, operator, or a passenger,in the vehicle. Any device herein may be mounted onto, may be attachedto, may be part of, or may be integrated in, the vehicle. Any vehicleherein may be a ground vehicle adapted to travel on land, and may beselected from the group consisting of a bicycle, a car, a motorcycle, atrain, an electric scooter, a subway, a train, a trolleybus, and a tram.Further, any ground vehicle herein may consist of, or may comprise, anautonomous car. Any autonomous car herein may be according to levels 0,1, or 2 of the Society of Automotive Engineers (SAE) J3016 standard, ormay be according to levels 3, 4, or 5 of the Society of AutomotiveEngineers (SAE) J3016 standard. Any vehicle herein may be a buoyant orsubmerged watercraft adapted to travel on or in water, and anywatercraft herein may be selected from the group consisting of a ship, aboat, a hovercraft, a sailboat, a yacht, and a submarine. Alternativelyor in addition, any vehicle herein may be an aircraft adapted to fly inair, and any aircraft herein may be a fixed wing or a rotorcraftaircraft, and may be selected from the group consisting of an airplane,a spacecraft, a glider, a drone, or an Unmanned Aerial Vehicle (UAV).

Any receiving herein of any video data stream may comprise receiving ofthe video data stream over a wireless network by a first wirelesstransceiver via a first antenna from the video camera. Further, anyreceiving herein of any second signal may comprise receiving of thesecond signal over a wireless network by a first wireless transceivervia a first antenna from the sensor. Any method herein may furthercomprise transmitting the video data stream over a wireless network by asecond wireless transceiver via a second antenna from the video camera.Further, any method herein may further comprise transmitting of thesecond signal over a wireless network by a second wireless transceivervia a second antenna from the sensor. Any method herein may be used witha received multiplexed signal that may comprise the video data streamand the second signal, and any method herein may further comprisede-multiplexing the received multiplexed signal into the video datastream and the second signal. Furthermore, any method herein may furthercomprise multiplexing the video data stream and the second signal intothe received multiplexed signal, and any multiplexing herein may bebased on, or may use, Frequency Division/Domain Multiplexing (FDM) orTime Domain/Division Multiplexing (TDM).

Any wireless network herein may comprise a Wireless Wide Area Network(WWAN), any wireless transceiver herein may comprise a WWAN transceiver,and any antenna herein may comprise a WWAN antenna. Any WWAN herein maybe a wireless broadband network. Any WWAN herein may be a WiMAX network,any antenna herein may be a WiMAX antenna and any wireless transceiverherein may be a WiMAX modem, and the WiMAX network may be according to,compatible with, or based on, Institute of Electrical and ElectronicsEngineers (IEEE) IEEE 802.16-2009. Alternatively or in addition, theWWAN may be a cellular telephone network, any antenna herein may be acellular antenna, and any wireless transceiver herein may be a cellularmodem, where the cellular telephone network may be a Third Generation(3G) network that uses Universal Mobile Telecommunications System(UMTS), Wideband Code Division Multiple Access (W-CDMA) UMTS, High SpeedPacket Access (HSPA), UMTS Time-Division Duplexing (TDD), CDMA20001×RTT, Evolution—Data Optimized (EV-DO), or Global System for Mobilecommunications (GSM), Enhanced Data rates for GSM Evolution (EDGE)EDGE-Evolution, or the cellular telephone network may be a FourthGeneration (4G) network that uses Evolved High Speed Packet Access(HSPA+), Mobile Worldwide Interoperability for Microwave Access (WiMAX),Long-Term Evolution (LTE), LTE-Advanced, Mobile Broadband WirelessAccess (MBWA), or is based on IEEE 802.20-2008.

Any wireless network herein may comprise a Wireless Personal AreaNetwork (WPAN), any wireless transceiver herein may comprise a WPANtransceiver, and any antenna herein may comprise an WPAN antenna. TheWPAN may be according to, compatible with, or based on, Bluetooth™,Bluetooth Low Energy (BLE), or IEEE 802.15.1-2005 standards, or the WPANmay be a wireless control network that may be according to, or may bebased on, Zigbee™, IEEE 802.15.4-2003, or Z-Wave™ standards. Anywireless network herein may comprise a Wireless Local Area Network(WLAN), any wireless transceiver herein may comprise a WLAN transceiver,and any antenna herein may comprise a WLAN antenna. The WLAN may beaccording to, may be compatible with, or may be based on, a standardselected from the group consisting of IEEE 802.11-2012, IEEE 802.11a,IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, and IEEE 802.11ac. Anywireless network herein may be over a licensed or unlicensed radiofrequency band that may be an Industrial, Scientific and Medical (ISM)radio band.

Any wireless network herein may be using, or may be based on, DedicatedShort-Range Communication (DSRC) that may be according to, may becompatible with, or may be based on, European Committee forStandardization (CEN) EN 12253:2004, EN 12795:2002, EN 12834:2002, EN13372:2004, or EN ISO 14906:2004 standard. Alternatively or in addition,the DSRC may be according to, may be compatible with, or may be basedon, IEEE 802.11p, IEEE 1609.1-2006, IEEE 1609.2, IEEE 1609.3, IEEE1609.4, or IEEE1609.5.

Any enclosure herein, such as each of the first enclosure and the secondenclosure may be permanently or releaseably attachable to, or may bepart of, a clothing piece of a person. The attaching may use taping,gluing, pinning, enclosing, encapsulating, a pin, or a latch and hookclip, and the clothing piece may be a top, bottom, or full-bodyunderwear, or a headwear, a footwear, an accessory, an outwear, a suit,a dress, a skirt, or a top. Any enclosure herein, such as the thirdenclosure, may be a portable or a hand-held enclosure. Any power sourceherein, such as the third power source, may be a primary or rechargeablebattery.

Any device, component, or apparatus herein, such as the third device orthe component, may consist of, or may comprise, a notebook, a laptopcomputer, a media player, a cellular phone, a smartphone, a PersonalDigital Assistant (PDA), and may comprise a memory for storing software,and a processor for executing the software. Any smartphone herein mayconsist of, may be based on, or may comprise, an Apple iPhone 6 or aSamsung Galaxy S6. Any software herein may comprise an operating systemthat may be a mobile operating system, which may comprise, may use, ormay be based on, Android version 2.2 (Froyo), Android version 2.3(Gingerbread), Android version 4.0 (Ice Cream Sandwich), Android Version4.2 (Jelly Bean), Android version 4.4 (KitKat)), Apple iOS version 3,Apple iOS version 4, Apple iOS version 5, Apple iOS version 6, Apple iOSversion 7, Microsoft Windows® Phone version 7, Microsoft Windows® Phoneversion 8, Microsoft Windows® Phone version 9, or Blackberry® operatingsystem.

Any digital camera herein may comprise an optical lens for focusingreceived light, the lens may be mechanically oriented to guide thecaptured images; a photosensitive image sensor array that may bedisposed approximately at an image focal point plane of the optical lensfor capturing the image and producing an analog signal representing theimage; and an analog-to-digital (A/D) converter that may be coupled tothe image sensor array for converting the analog signal to a digitaldata representation of the captured image.

The image sensor array may be operative to respond to visible ornon-visible light, such as infrared, ultraviolet, X-rays, or gamma rays.The image sensor array may use, or may be based on, semiconductorelements that may use the photoelectric or photovoltaic effect. Theimage sensor array may use, may comprise, or may be based on,Charge-Coupled Devices (CCD) or Complementary Metal-Oxide-SemiconductorDevices (CMOS) elements.

Any digital camera herein may further comprise an image processor thatmay be coupled to the image sensor array for providing a digital videodata signal according to a digital video format, the digital videosignal may be carrying digital data video that may comprise, may becompatible with, or may be based on, the captured images. Any digitalvideo format herein may use, may be compatible with, or may be based onTIFF (Tagged Image File Format), RAW format, AVI, DV, MOV, WMV, MP4, DCF(Design Rule for Camera Format), ITU-T H.261, ITU-T H.263, ITU-T H.264,ITU-T CCM 601, ASF, Exif (Exchangeable Image File Format), or DPOF(Digital Print Order Format) standards. Any digital camera herein mayfurther comprise a video compressor coupled to the image sensor arrayfor compressing the digital data video, and the video compressor mayperform a compression scheme that may use, or may be based on,intraframe or interframe compression. The compression scheme may belossy or non-lossy, and may use, may be compatible with, or may be basedon, JPEG (Joint Photographic Experts Group) and MPEG (Moving PictureExperts Group), ITU-T H.261, ITU-T H.263, ITU-T H.264, or ITU-T CCM 601.

Any system, device, component, or apparatus herein may be used with awireless network. The first, second, and third devices may respectivelycomprise a first, second, and third wireless transceivers and maycommunicate over the wireless network respectively using the first,second, and third wireless transceivers. The first wireless transmittermay be part of the first wireless transceiver, the second wirelesstransmitter may be part of the second wireless transceiver, and thewireless receiver may be part of the third wireless transceiver.

Any system, device, component, or apparatus herein may be used with avehicle operative to travel in a travel direction under control of adriver. Any system, device, component, or apparatus herein may bemountable on, attachable to, or part of, the vehicle. The third antennaor any antenna, the third device or any device herein, or any otherelement herein may be mountable on, attachable to, or part of, thevehicle, and may be located along the travel direction or along a lineof sight to the road of the driver. Any component, device, or apparatusherein may consist of, may comprise, may be integrated with, may beconnectable to, or may be part of, an Electronic Control Unit (ECU),which may be an Electronic/engine Control Module (ECM), Engine ControlUnit (ECU), Powertrain Control Module (PCM), Transmission Control Module(TCM), Brake Control Module (BCM or EBCM), Central Control Module (CCM),Central Timing Module (CTM), General Electronic Module (GEM), BodyControl Module (BCM), Suspension Control Module (SCM), Door Control Unit(DCU), Electric Power Steering Control Unit (PSCU), Seat Control Unit,Speed Control Unit (SCU), Telematic Control Unit (TCU), TransmissionControl Unit (TCU), Brake Control Module (BCM; ABS or ESC), Batterymanagement system, control unit, or a control module.

Any vehicle herein may further comprise an Advanced Driver AssistanceSystems (ADAS) functionality, system, or scheme, and any device, system,component, or apparatus herein, such as the third device, may be partof, integrated with, communicating with, or coupled to, the ADASfunctionality, system, or scheme. The ADAS functionality, system, orscheme may consist of, may comprise, or may use, Adaptive Cruise Control(ACC), Adaptive High Beam, Glare-free high beam and pixel light,Adaptive light control such as swiveling curve lights, Automaticparking, Automotive navigation system with typically GPS and TMC forproviding up-to-date traffic information, Automotive night vision,Automatic Emergency Braking (AEB), Backup assist, Blind Spot Monitoring(BSM), Blind Spot Warning (BSW), Brake light or traffic signalrecognition, Collision avoidance system, Pre-crash system, CollisionImminent Braking (CIB), Cooperative Adaptive Cruise Control (CACC),Crosswind stabilization, Driver drowsiness detection, Driver MonitoringSystems (DMS), Do-Not-Pass Warning (DNPW), Electric vehicle warningsounds used in hybrids and plug-in electric vehicles, Emergency driverassistant, Emergency Electronic Brake Light (EEBL), Forward CollisionWarning (FCW), Heads-Up Display (HUD), Intersection assistant, Hilldescent control, Intelligent speed adaptation or Intelligent SpeedAdvice (ISA), Intelligent Speed Adaptation (ISA), Intersection MovementAssist (IMA), Lane Keeping Assist (LKA), Lane Departure Warning (LDW)(a.k.a. Line Change Warning—LCW), Lane change assistance, Left TurnAssist (LTA), Night Vision System (NVS), Parking Assistance (PA),Pedestrian Detection System (PDS), Pedestrian protection system,Pedestrian Detection (PED), Road Sign Recognition (RSR), Surround ViewCameras (SVC), Traffic sign recognition, Traffic jam assist, Turningassistant, Vehicular communication systems, Autonomous Emergency Braking(AEB), Adaptive Front Lights (AFL), or Wrong-way driving warning.

Video files that are received from aerial platforms may incorporatetelemetries stream describing the position, orientation, or motion ofthe aircraft and camera, for the purpose of status report and controlover the equipment by remote operator. The correlation between the twoinformation sources, namely visual and telemetries, may be utilized.Visual may be visible light video, other bandwidth video (IR, thermal,radio imaging, CAT scan, etc.), ELOP imagery (LIDAR, SONAR, RADAR etc.).Telemetry may include any information regarding the visual source state,such as its position, speed, acceleration, temperature etc.

The streams may be time-synchronized to each other or not. If they aresynchronized, the level of synchronization, such as the time gap betweenthe representations of a specific event in the two streams, may vary.The telemetries and video streams may be loosely synchronized, may bereported at different frequencies, and may be usually reported with atime offset between them, where the time difference may be up to severalseconds in magnitude. In order to use the telemetries stream foraugmenting virtual features over the video, the telemetries should betime-synchronized to the video as closely as possible, for allowing thetime offset should be minimized.

The correlated information may include changes to the video source,camera position, camera velocity, camera acceleration, FOV (Field ofView) or Zoom, payload operation (such as moving from one camera toanother or moving from visible to IR sensor), satellite navigationsystem (such as GPS) reception level, ambient light level, wind speed(such as identifying wind gusts from movement of trees in the capturedvideo), or vibrations. The correlation may use time spans during whichthe telemetries and video do not change, based on their length.

The correlation between information in the two streams may be used toachieve high quality synchronization and low-error time alignment)between them. For example, the system may correlate changes withoutrelating to their specific nature, such as allowing any video abruptchange to correlate to any telemetry abrupt change. Further, a methodmay comprise analyzing the video and identifying an aircraft and acamera motion in it, finding corresponding motion reports in thetelemetries stream, and using the correspondence to derive the timeoffset between the two streams. The resulted information may be used toalign the video and telemetries streams so that the time difference isminimized. The system could attempt to re-synch the streams with any newchange detected, at a given frequency, when it is suspected that thestreams are not aligned, or only once. The previous alignment could bethe used for next alignment search algorithm. Corresponding motionscould be, for example, rotation of the aircraft, zoom in of the camera,weight-on-wheels of the aircraft, or switching between camera sensors.Further, the video and telemetries with minimized time offset may beused for augmenting (virtual) features to the video stream.

Any method or system herein may be used for adding Augmented Realitylayers over video, both in real time and after the fact, for vehiclenavigation, where vehicle could be airborne, terrestrial, marine orsubmarine, for analysis of the communication channels carrying the Videoand telemetry, i.e for detection of introduced latency and tampering,for aligning video and telemetries for later analysis of recordedmaterial, i.e inspection, surveying, monitoring in agriculture, mining,building, or for 3D object analysis (such as SLAM or in dronedeployment).

A tangible machine-readable medium (such as a storage) may have a set ofinstructions detailing part (or all) of the methods and steps describedherein stored thereon, so that when executed by one or more processors,may cause the one or more processors to perform part of, or all of, themethods and steps described herein. Any of the network elements may be acomputing device that comprises a processor and a computer-readablememory (or any other tangible machine-readable medium), and thecomputer-readable memory may comprise computer-readable instructionssuch that, when read by the processor, the instructions causes theprocessor to perform the one or more of the methods or steps describedherein. A non-transitory computer readable medium containing computerinstructions that, when executed by a computer processor, cause theprocessor to perform at least part of the steps described herein.

The above summary is not an exhaustive list of all aspects of thepresent invention. Indeed, it is contemplated that the inventionincludes all systems and methods that can be practiced from all suitablecombinations and derivatives of the various aspects summarized above, aswell as those disclosed in the detailed description below andparticularly pointed out in the claims filed with the application. Suchcombinations have particular advantages not specifically recited in theabove summary.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of non-limiting examples only,with reference to the accompanying drawings, wherein like designationsdenote like elements. Understanding that these drawings only provideinformation concerning typical embodiments and are not therefore to beconsidered limiting in scope:

FIG. 1 illustrates schematically a simplified schematic block diagram ofa prior-art digital video camera;

FIG. 2 depicts pictorially definitions of an aircraft axes and motionaround the axes;

FIG. 2a illustrates a table of the various classification levels ofautonomous car is according to the Society of Automotive Engineers (SAE)J3016 standard;

FIG. 3a depicts schematically an HMD for a VR application;

FIG. 3b depicts schematically a person wearing an HMD;

FIG. 4 illustrates schematically a prior-art simplified schematic blockdiagram arrangement of capturing an image by a digital video camera andusing the resulted video data stream;

FIG. 5 illustrates schematically a simplified schematic block diagramarrangement of estimating and using a delay associated with a video datastream based on physical phenomenon affecting a sensor and a camera;

FIG. 5a illustrates schematically a simplified schematic block diagramarrangement of estimating and using a delay associated with a video datastream based on physical phenomenon affecting a sensor and a scenecaptured by a camera;

FIG. 6 illustrates schematically a simplified schematic block diagramarrangement of a single enclosure that includes all components forestimating and using a delay associated with a video data stream;

FIGS. 6a, 6b, and 6c illustrate schematically a simplified schematicblock diagram arrangement of two enclosures that include components forestimating and using a delay associated with a video data stream;

FIGS. 7 and 7 a illustrate schematically two compared graphscorresponding to two signals; and

FIG. 8 illustrates schematically a simplified schematic block diagramarrangement of two enclosures communicating over a wireless network.

DETAILED DESCRIPTION

The principles and operation of an apparatus or a method according tothe present invention may be understood with reference to the figuresand the accompanying description wherein identical or similar components(either hardware or software) appearing in different figures are denotedby identical reference numerals. The drawings and descriptions areconceptual only. In actual practice, a single component can implementone or more functions; alternatively or in addition, each function canbe implemented by a plurality of components and devices. In the figuresand descriptions, identical reference numerals indicate those componentsthat are common to different embodiments or configurations. Identicalnumerical references (in some cases, even in the case of using differentsuffix, such as 5, 5 a, 5 b and 5 c) refer to functions or actualdevices that are either identical, substantially similar, similar, orhaving similar functionality. It is readily understood that thecomponents of the present invention, as generally described andillustrated in the figures herein, could be arranged and designed in awide variety of different configurations. Thus, the following moredetailed description of the embodiments of the apparatus, system, andmethod of the present invention, as represented in the figures herein,is not intended to limit the scope of the invention, as claimed, but ismerely representative of embodiments of the invention. It is to beunderstood that the singular forms “a”, “an”, and “the” herein includeplural referents unless the context clearly dictates otherwise. Thus,for example, a reference to “a component surface” includes a referenceto one or more of such surfaces. By the term “substantially” it is meantthat the recited characteristic, parameter, feature, or value need notbe achieved exactly, but that deviations or variations, including, forexample, tolerances, measurement error, measurement accuracy limitationsand other factors known to those of skill in the art, may occur inamounts that do not preclude the effect the characteristic was intendedto provide.

All directional references used herein (e.g., upper, lower, upwards,downwards, left, right, leftward, rightward, top, bottom, above, below,vertical, horizontal, clockwise, and counterclockwise, etc.) are onlyused for identification purposes to aid the reader's understanding ofthe present invention, and do not create limitations, particularly as tothe position, orientation, or use of the invention. Spatially relativeterms, such as “inner,” “outer,” “beneath”, “below”, “right”, “left”,“upper”, “lower”, “above”, “front”, “rear”, “left”, “right” and thelike, may be used herein for ease of description to describe one elementor feature's relationship to another element(s) or feature(s) asillustrated in the figures. Spatially relative terms may be intended toencompass different orientations of the device in use or operation inaddition to the orientation depicted in the figures. For example, if thedevice in the figures is turned over, elements described as “below” or“beneath” other elements or features would then be oriented “above” theother elements or features. Thus, the example term “below” can encompassboth an orientation of above and below. The device may be otherwiseoriented (rotated 90 degrees or at other orientations) and the spatiallyrelative descriptors used herein interpreted accordingly.

The term ‘Horizontal’ herein refers to include a direction, line,surface, or plane that is parallel or substantially parallel to theplane of the horizon. The term ‘substantially horizontal’ includes adirection, line, surface, or plane that is forming an angle of less than20°, 18°, 15°, 13°, 10°, 8°, 5°, 3°, 2°, 1°, 0.8°, 0.5°, 0.3°, 0.2°, or0.1° from an ideal horizontal line. The term ‘Vertical’ herein refers toinclude a direction, line, surface, or plane that is an upright orparallel or at right angles to a horizontal plane. The term‘substantially vertical’ includes a direction, line, surface, or planethat is forming an angle of less than 20°, 18°, 15°, 13°, 10°, 8°, 5°,3°, 2°, 1°, 0.8°, 0.5°, 0.3°, 0.2°, or 0.1° from an ideal vertical.

All directional references used herein (e.g., upper, lower, upwards,downwards, left, right, leftward, rightward, top, bottom, above, below,vertical, horizontal, clockwise, and counterclockwise, etc.) are onlyused for identification purposes to aid the reader's understanding ofthe present invention, and do not create limitations, particularly as tothe position, orientation, or use of the invention.

An example of an arrangement 50 that may be used to estimate or measurean actual delay of a video data stream is shown in FIG. 5, and is basedon the arrangement 40 shown in FIG. 4. The arrangement 50 includes aphysical phenomenon 54, pictorially illustrated as a ‘star’ shape. Asensor 51 may be used for sensing the physical phenomenon 54, and mayproduce an output signal f2(t) 55 b that represents, or is associatedwith, the time changing characteristic or value of the physicalphenomenon 54. For example, output signal f2(t) 55 b may be ameasurement, an estimation, or an indication of the status or magnitudeof the sensed or detected physical phenomenon 54. It is assumed that thedelay or latency induced from the actual physical phenomenon affect orvalue until outputted by the sensor 51 is negligible, and is thusassumed zero.

The video camera 10 and the sensor 51 are assumed to be both affectedsimilarly or differently by the physical phenomenon 54, as illustratedby respective dashed lines 54 a and 54 b. For example, the video camera10 and the sensor 51 may be mechanically attached (or otherwise coupled)so that both may sense a mechanical related physical phenomenon 54, orthey may be in the same location or in vicinity of each other (such asin a same enclosure) so they are both affected by the same physicalphenomenon 54.

A video processor 52 received the video data stream 43 and applies anvideo processing algorithm for extracting the effect of the physicalphenomenon 54, and produces a signal 55 a f′2(t-Δt) that represents theeffect of the physical phenomenon 54. In one example, the signal 55 af′2(t-Δt) may comprise an estimation, a calculation, or an indication ofthe change in time of a magnitude or value of the physical phenomenon54, based on analyzing the video data stream 43. Preferably, the outputsignal 55 a f′2(t-Δt) may be similar to the representation of thephysical phenomenon 54 as sensed by the sensor 51 and represented by thesignal f2(t) 55 b that is output from the sensor 51. The time delay Δtrepresents the delay at the video processor 52 output from the actualoccurrence of the physical phenomenon 54. In one example, the delayinduced by the video processor 52 itself in ineligible, thus can beassumed as zero, so that Δt=Δ′t. In another example, the delay inducedby the video processor 52 itself is fixed in time, such as δt, so thatΔt=Δ′t+δt. In any case, the value of Δt may be used to calculate orestimate Δ′t, such as by Δ′t=Δt−δt.

In one example, the output signals f′2(t-Δt) 55 a and f2(t) 55 b areidentical or somewhat similar, since they are both derived from, areassociated with, or represents in some way, the same physical phenomenon54, with the exception of one signal being delayed by Δt from the other.A comparator 53 compares the two signals to each other, and estimatesthe delay time Δt 56 at its output. The estimated delay time Δt 56 maybe fed to the combiner 41, to be used therein for synchronizing theadditional data 42 with the video data stream 43, for obtaininglow-error time alignment at the output 44.

The video camera 10 may correspond to the digital camera shown in FIG.1, and may comprises an optical lens for focusing received light, thelens being mechanically oriented to guide a captured image; aphotosensitive image sensor array disposed approximately at an imagefocal point plane of the optical lens for capturing the image andproducing an analog signal representing the image; and ananalog-to-digital (A/D) converter coupled to the image sensor array forconverting the analog signal to the digital video stream. Further, theimage sensor array may operative to respond to a visible or non-visiblelight, which may be infrared, ultraviolet, X-rays, or gamma rays. Theimage sensor array may comprise, use, or may be based on, semiconductorelements that use the photoelectric or photovoltaic effect, such asCharge-Coupled Devices (CCD) or Complementary Metal-Oxide-SemiconductorDevices (CMOS) elements.

Further, the digital video camera further comprises an image processorcoupled to the image sensor array for providing the video data streamaccording to a digital video format, that may use, may be compatiblewith, or may be based on, TIFF (Tagged Image File Format), RAW format,AVI, DV, MOV, WMV, MP4, DCF (Design Rule for Camera Format), ITU-TH.261, ITU-T H.263, ITU-T H.264, ITU-T CCM 601, ASF, Exif (ExchangeableImage File Format), or DPOF (Digital Print Order Format) standards.Further, the video data stream may be in a High-Definition (HD) orStandard-Definition (SD) format, and may further be based on, may becompatible with, or may be according to, ISO/IEC 14496 standard, MPEG-4standard, or ITU-T H.264 standard.

In one example, the time delay value Δt 56 is estimated continuously bythe comparator 53, providing a continuous and time-changing updatedvalue to be used by the combiner 41. Alternatively or in addition, thetime delay value Δt 56 may be estimated once or in response to an event,and thereafter last estimated value is used by the combiner 41. Forexample, the time delay value Δt 56 may be estimated as part of apower-up process (or software boot process) of a device or upon arequest or a user, and the unchanged value is used afterwards.Alternatively or in addition, the time delay value Δt 56 may beperiodically estimated, where an estimated value is used until the nextestimation process is concluded, and then the updated value is used. Thetime period between consecutive value estimation may be equal to, lessthan, or more than 1 second, 2 seconds, 5 seconds, 10 seconds, 20seconds, 30 seconds, 1 minute, 2, minutes, 5 minutes, 10 minutes, 20minutes, 30 minutes, 1 hour, 2 hours, 5 hours, 10 hours, 1 day, 2 days,4 days, 1 week, 2 weeks, 3 weeks, or 1 months. The periodically orcontinuously updating the estimation of the time delay value Δt 56 maybe useful in configurations where the delay Δ′t may change over time dueto environmental or operational changes in the system.

Any technique or method for estimating the time delay value Δt 56 may beused by the comparator 53. In one example, a convolution or correlationtechnique is used for comparing the two incoming signals f′2(t-Δt) 55 aand f2(t) 55 b for estimating the resulting time delay value Δt 56. Across-correlation operation may be used, where one of signals is timeshifted using a varied delay, and a cross-correlation coefficient of thetime shifted signal and the second incoming signals is calculated. Thetime shift that results in the maximum value of the cross-correlationcoefficient, such as by using peak-detection mechanism, may be used asthe estimated time delay value Δt 56.

Alternatively or in addition, the comparator 53 may use any of thevarious methods for Time-Delay Estimation (TDE) in linear dynamicsystems that are described in a thesis by “A Survey and Comparison ofTime-Delay Estimation Methods in Linear Systems” by Svante Bjorklund,published 2003 by the Department of Electrical Engineering, LinkopingsUniversitet, Linkoping, Sweden [ISBN-91-7373-870-0], or the comparator53 may use any of the various methods for Time-Delay Estimation (TDE) inlinear dynamic systems that are described in a Report no.:LiTH-ISY-R-2554 entitled: “A Review of Time-Delay Estimation Techniques”and published Dec. 30, 2003, by Svante Bjorklund and Lennart Ljung ofthe Control & Communication, Department of Electrical Engineering,Linkopings Universitet, SE-58183 Linkoping, Sweden, which are bothincorporated in their entirety for all purposes as if fully set forthherein. A classification according to underlying principles of TDEmethods is described, where the main classes are: 1) Time-DelayApproximation Methods: The time-delay is estimated from a relation (amodel) between the input and output signals expressed in a certainbasis. The time delay is not an explicit parameter in the model. 2)Explicit Time-Delay Parameter Methods: The time-delay is an explicitparameter in the model. 3) Area and Moment Methods: The time-delay isestimated from certain integrals of the impulse and step responses. 4)Higher Order Statistics Methods.

Alternatively or in addition, the comparator 53 may use any of thevarious techniques which are useful for time alignment that aredescribed in a paper entitled: “Time Alignment Techniques forExperimental Sensor Data” by Matthew Rhudy of Lafayette College, Easton,Pa., 18042, USA [DOI:10.5121/ijcses.2014.5201], presented on April 2014at International Journal of Computer Science & Engineering Survey(IJCSES) Vol. 5, No. 2, which is incorporated in its entirety for allpurposes as if fully set forth herein.

Alternatively or in addition, the comparator 53 may use any of the fivedifferent time delay estimation methods are described in a paperentitled “A Comparative Study of Time-Delay Estimation Techniques UsingMicrophone Arrays” by Yushi Zhang and Waleed H. Abdulla, published 2005as ‘School of Engineering Report No. 619’ by Department of Electricaland Computer Engineering, The University of Auckland, Private Bag 92019,Auckland, New Zealand, which is incorporated in its entirety for allpurposes as if fully set forth herein. These methods arecross-correlation (CC), phase transform (PHAT), maximum likelihoodestimator (ML), adaptive least mean square filter (LMS) and averagesquare difference function (ASDF). Their simulation results are comparedin terms of computation complexity, hardware implementation, precision,and accuracy. Since the performances of the TDE methods are considerablydegraded by the signal-to-noise ratio (SNR) level, this factor has beentaken as a prime factor in benchmarking the different methods.

Alternatively or in addition, the comparator 53 may use the MATLABfunction ‘finddelay’ used for estimating delay(s) between signals,described in https://www.mathworks.com/help/signal/ref/finddelay.htmldownloaded from the Internet on January 2019 provided by The MathWorks®,Inc., which is incorporated in its entirety for all purposes as if fullyset forth herein.

Alternatively or in addition, the comparator 53 may use, or may be basedon, a system and process for estimating the time delay of arrival (TDOA)between a pair of audio sensors of a microphone array, which ispresented in U.S. Pat. No. 7,113,605 to Rui et al. entitled: “System andprocess for time delay estimation in the presence of correlated noiseand reverberation”, which is incorporated in its entirety for allpurposes as if fully set forth herein. Generally, a generalizedcross-correlation (GCC) technique is employed. However, this techniqueis improved to include provisions for both reducing the influence(including interference) from correlated ambient noise and reverberationnoise in the sensor signals prior to computing the TDOA estimate. Twounique correlated ambient noise reduction procedures are also proposed.One involves the application of Wiener filtering, and the other acombination of Wiener filtering with a Gnn subtraction technique. Inaddition, two unique reverberation noise reduction procedures areproposed. Both involve applying a weighting factor to the signals priorto computing the TDOA which combines the effects of a traditionalmaximum likelihood (TML) weighting function and a phase transformation(PHAT) weighting function.

In one example, the two incoming signals f′2(t-Δt) 55 a and f2(t) 55 bare carried or represented as analog or digital electrical signals.Alternatively or in addition, then the comparator 53 may use, or may bebased on, evaluating a delay between the first signal and the secondsignal by deriving from the first signal substantially aperiodic events,possibly by using a zero-crossing detector on a random signal, and usingthese events to define respective segments of the second signal, aspresented in U.S. Pat. No. 6,539,320 to Szajnowski et al. entitled:“Time delay determination and determination of signal shift”, which isincorporated in its entirety for all purposes as if fully set forthherein. The segments are combined, e.g., by averaging, to derive awaveform which includes a feature representing coincidences of parts ofthe second signal associated with the derived events. The delay isdetermined from the position within the waveform of this feature.

Alternatively or in addition, the comparator 53 may use, or may be basedon, plural independent delay lines are connected in parallel and receivea first signal taken from a first point, as described in U.S. Pat. No.4,779,215 to Moisan et al. entitled: “Method and device for measuring bycorrelation, in real time, the delays between matching electricalsignals”, which is incorporated in its entirety for all purposes as iffully set forth herein. Samples of the first signal are taken atdifferent sampling frequencies and are respectively stored in theseveral delay lines. Samples of a second signal are taken from a secondpoint and are stored in a direct line at one of the frequencies. Thecorrelation function between the samples of the direct line and thesamples of one of the delay lines is calculated. Exemplary applicationsinclude measuring the speed of a linearly translating object, such as aweb of paper.

It may practically beneficial to compare the two incoming signalsf′2(t-Δt) 55 a and f2(t) 55 b for estimating the resulting time delayvalue Δt 56 during a pre-set time-interval. Such time interval may beequal to, less than, or more than, 1 millisecond, 2 milliseconds, 5milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 50milliseconds, 100 milliseconds, 200 milliseconds, 500 milliseconds, 1second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1hour, 2 hours, 5 hours, or 10 hours.

Any timing information or time related measurements may use timers thatmay be implemented as a monostable circuit, producing a pulse of setlength when triggered. In one example, the timers are based on RC basedpopular timers such as 555 and 556, such as ICM7555 available from MaximIntegrated Products, Inc. of Sunnyvale, Calif., U.S.A., described in thedata sheet “General Purpose Timers” publication number 19-0481 Rev.211/92, which is incorporated in its entirety for all purposes as iffully set forth herein. Examples of general timing diagrams as well asmonostable circuits are described in Application Note AN170 “NE555 andNE556 Applications” from Philips semiconductors dated December 1988,which is incorporated in its entirety for all purposes as if fully setforth herein. Alternatively, a passive or active delay line may be used.Further, a processor based delay line can be used, wherein the delay isset by its firmware, typically as a service of the operation system.

The comparator 53 may be based on comparing timings of two correspondingdiscrete events respectively in the two incoming signals f′2(t-Δt) 55 aand f2(t) 55 b, for estimating the resulting time delay value Δt 56during a pre-set time-interval.

Such an example is shown in a view 70 shown in FIG. 7. The lower graphincludes a magnitude axis 71 b versus a time axis 72, showing amagnitude 73 b (such as an amplitude) of the f2(t) signal 55 b. Thehigher graph includes a magnitude axis 71 a versus the same time axis72, showing a magnitude 73 a (such as an amplitude) of the f′2(t) signal55 a. A defined event may be a peak detection in the signals. A peakvalue regarding the signal 73 b (corresponding to the signal f2(t) 55 b)is detected in a time point t1 74 a, and a peak value regarding thesignal 73 a (corresponding to the signal f′2(t) 55 a) is detected in atime point t2 74 b. In such a case, the estimated time delay Δt 56 maybe calculated or estimated as Δt=t2-t1.

In another example, the event may be associated with a pre-definedthreshold, as shown in a view 70 a shown in FIG. 7a . The lower graphincludes the magnitude axis 71 b versus the time axis 72, showing amagnitude 73 b (such as an amplitude) of the f2(t) signal 55 b. Thehigher graph includes the magnitude axis 71 a versus the same time axis72, showing the magnitude 73 a (such as an amplitude) of the f′2(t)signal that corresponds to signal f′2(t-Δt) 55 a in FIG. 5. A definedevent may be crossing a threshold value. A threshold value V2 75 b isassociated with the f2(t) signal 55 b, and the signal 73 b exceeds thethreshold V2 75 b at a time point t3 76 a. A threshold value V1 75 a isassociated with the f′2(t) signal 55 a, and the signal 73 a exceeds thethreshold V1 75 a at a time point t4 76 b. In such a case, the estimatedtime delay Δt 56 may be calculated or estimated as Δt=t4-t3.

The peak detection mechanism as shown in the view 70 in FIG. 7, or thethreshold crossing mechanism as shown in the view 70 a in FIG. 7a , maybe applied directly to the signal at the output of the video processor52, or to the sensor 51 output, or to both. Further, any other signalanalysis may be equally applied. Alternatively or in addition, the peakdetection mechanism, the threshold crossing mechanism, or any other timedelay estimation mechanism, may be applied to any function of the videoprocessor 52, to the sensor 51 output, or to both.

Any function, discrete or continuous, monotonic or non-monotonic, may beused to manipulate the video processor 52, the sensor 51 output, orboth, before applying any time delay estimation or comparing. Thefunction may be an elementary function that is built from basicoperations (e.g. addition, exponentials, and logarithms) such as anAlgebraic function that can be expressed as the solution of a polynomialequation with integer coefficients, Polynomials that may be addition,multiplication, and exponentiation, such as Linear function (Firstdegree polynomial, graph is a straight line), Quadratic function (Seconddegree polynomial, graph is a parabola), Cubic function (Third degreepolynomial), Quartic function (Fourth degree polynomial), Quinticfunction (Fifth degree polynomial), Sextic function (Sixth degreepolynomial), or Rational functions (A ratio of two polynomials).Similarly, the function may be an Nth root based, such as a Square rootor a Cube root. Alternatively or in addition, a non-algebraic functionmay be used, such as a Transcendental function, that may be Exponentialfunction that raises a fixed number to a variable power, Hyperbolicfunctions that uses trigonometric functions, Logarithmic function, or aPower function that raises a variable number to a fixed power. Thefunction may be a periodic function such as a trigonometric functions,that may use or include sine, cosine, tangent, cotangent, secant,cosecant, exsecant, excosecant, versine, coversine, vercosine,covercosine, haversine, hacoversine, havercosine, or hacovercosine,typically used in geometry.

Time-frequency Analysis. A time-frequency analysis comprises thosetechniques that study a signal in both the time and frequency domainssimultaneously, using various time-frequency representations. Ratherthan viewing a 1-dimensional signal (a function, real or complex-valued,whose domain is the real line) and some transform (another functionwhose domain is the real line, obtained from the original via sometransform), time-frequency analysis studies a two-dimensional signal—afunction whose domain is the two-dimensional real plane, obtained fromthe signal via a time-frequency transform. Time-Frequency analysis isdescribed in an article by Rolf Hut (September 2004) entitled: “TimeFrequency Analysis—a Comparison between cochlear modeling and existingmethods”, and in an article by Franz Hlawatsch and Gerald Matz (of theInstitute of Communications and radio-Frequency Engineering, ViennaUniversity of Technology) entitled: “Time-Frequency Signal Processing: AStatistical Perspective”, which are both incorporated in their entiretyfor all purposes as if fully set forth herein. One of the most basicforms of time-frequency analysis is the Short-Time Fourier Transform(STFT), but more sophisticated techniques have been developed, such aswavelets.

There are several different ways to formulate a valid time-frequencydistribution function, resulting in several well-known time-frequencydistributions, such as: Short-time Fourier transform (including theGabor transform); Wavelet transform; Bilinear time-frequencydistribution function (Wigner distribution function, or WDF); andModified Wigner distribution function or Gabor-Wigner distributionfunction.

Alternatively or in addition, a frequency-domain analysis is performedbefore applying further mechanism, such as peak detection or thresholdcrossing. A frequency-domain representation can also include informationon the phase shift that must be applied to each sinusoid in order to beable to recombine the frequency components to recover the original timesignal. An example for such conversion may be the Fourier transform,which converts the time-function into a sum of sine waves of differentfrequencies, each of which represents a frequency component. The‘spectrum’ of frequency components is the frequency domainrepresentation of the signal. The inverse Fourier transform converts thefrequency domain function back to a time function. A spectrum analyzeris the tool commonly used to visualize real-world signals in thefrequency domain. Some specialized signal processing techniques usetransforms that result in a joint time-frequency domain, with theinstantaneous frequency being a key link between the time domain and thefrequency domain.

There are a number of different mathematical transforms that may be usedto analyze time domain functions and are referred to as “frequencydomain” methods. The most common transforms are Fourier series, Fouriertransform, Laplace transform, Z transform, and Wavelet transform. TheFourier transform of a periodic signal only has energy at a basefrequency and its harmonics. Another way of saying this is that aperiodic signal can be analyzed using a discrete frequency domain.Dually, a discrete-time signal gives rise to a periodic frequencyspectrum. Combining these two, if we start with a time signal that isboth discrete and periodic, we get a frequency spectrum which is bothperiodic and discrete. This is the usual context for a discrete Fouriertransform. Converting to frequency domain as part of any“Frequency-Domain” herein may include, may use, or may be based on, oneor more of the methods described in articles by Boualem Boashashpublished in Proceedings of the IEEE, Vol. 80, No. 4, April 1992(0018-9219/92$03.00, 1992 IEEE) entitled: “Estimating and InterpretingThe Instantaneous Frequency of a Signal—Part 1: Fundamentals”, and“Estimating and Interpreting The Instantaneous Frequency of aSignal—Part 2: Algorithms and Applications”, and in an article byJonatan Lerga (of University of Rijeka) entitled: “Overview of SignalInstantaneous Frequency Estimation Methods”, which are all incorporatedin their entirety for all purposes as if fully set forth herein.

Any function, discrete or continuous, monotonic or non-monotonic, may beapplied to the signal at the output of the video processor 52, to thesensor 51 output, or to both. The function may be an elementary functionthat is built from basic operations (e.g. addition, exponentials, andlogarithms) such as an Algebraic function that can be expressed as thesolution of a polynomial equation with integer coefficients, Polynomialsthat may be addition, multiplication, and exponentiation, such as Linearfunction (First degree polynomial, graph is a straight line), Quadraticfunction (Second degree polynomial, graph is a parabola), Cubic function(Third degree polynomial), Quartic function (Fourth degree polynomial),Quintic function (Fifth degree polynomial), Sextic function (Sixthdegree polynomial), or Rational functions (A ratio of two polynomials).Similarly, the function may be an Nth root based, such as a Square rootor a Cube root. Alternatively or in addition, a non-algebraic functionmay be used, such as a Transcendental function, that may be Exponentialfunction that raises a fixed number to a variable power, Hyperbolicfunctions that uses trigonometric functions, Logarithmic function, or aPower function that raises a variable number to a fixed power. Thefunction may be a periodic function such as a trigonometric functions,that may use or include sine, cosine, tangent, cotangent, secant,cosecant, exsecant, excosecant, versine, coversine, vercosine,covercosine, haversine, hacoversine, havercosine, or hacovercosine,typically used in geometry.

Any element capable of measuring or responding to a physical phenomenonmay be used as the sensor 51. An appropriate sensor may be adapted for aspecific physical phenomenon, such as a sensor responsive totemperature, humidity, pressure, audio, vibration, light, motion, sound,proximity, flow rate, electrical voltage, and electrical current. Thesensor 51 may be an analog sensor having an analog signal output such asanalog voltage or current, or may have continuously variable impedance.Alternatively on in addition, the sensor 51 may have a digital signaloutput. The sensor 51 may serve as a detector, notifying only thepresence of a phenomenon, such as by a switch, and may use a fixed orsettable threshold level. The sensor 51 may measure time-dependent orspace-dependent parameters of a phenomenon. The sensor 51 may measuretime-dependencies or a phenomenon such as the rate of change,time-integrated or time-average, duty-cycle, frequency or time periodbetween events. The sensor 51 may be a passive sensor, or an activesensor requiring an external source of excitation. The sensor 51 may besemiconductor-based, and may be based on MEMS technology.

The sensor 51 may measure the amount of a property or of a physicalquantity or the magnitude relating to a physical phenomenon, body orsubstance. Alternatively or in addition, a sensor may be used to measurethe time derivative thereof, such as the rate of change of the amount,the quantity or the magnitude. In the case of space related quantity ormagnitude, a sensor may measure the linear density, surface density, orvolume density, relating to the amount of property per volume.Alternatively or in addition, a sensor may measure the flux (or flow) ofa property through a cross-section or surface boundary, the fluxdensity, or the current. In the case of a scalar field, a sensor maymeasure the quantity gradient. A sensor may measure the amount ofproperty per unit mass or per mole of substance. A single sensor may beused to measure two or more phenomena.

The sensor 51 may provide an electrical output signal f2(t) 55 b inresponse to a physical, chemical, biological or any other phenomenon,serving as a stimulus to the sensor. The sensor may serve as, or be, adetector, for detecting the presence of the phenomenon. Alternatively orin addition, a sensor may measure (or respond to) a parameter of aphenomenon or a magnitude of the physical quantity thereof. For example,the sensor 51 may be a thermistor or a platinum resistance temperaturedetector, a light sensor, a pH probe, a microphone for audio receiving,or a piezoelectric bridge Similarly, the sensor 51 may be used tomeasure pressure, flow, force or other mechanical quantities. The sensor51 output may be amplified by an amplifier connected to the sensoroutput. Other signal conditioning may also be applied in order toimprove the handling of the sensor output or adapting it to the nextstage or manipulating, such as attenuation, delay, current or voltagelimiting, level translation, galvanic isolation, impedancetransformation, linearization, calibration, filtering, amplifying,digitizing, integration, derivation, and any other signal manipulation.Some sensors conditioning involves connecting them in a bridge circuit.In the case of conditioning, the conditioning circuit may added tomanipulate the sensor output, such as filter or equalizer for frequencyrelated manipulation such as filtering, spectrum analysis or noiseremoval, smoothing or de-blurring in case of image enhancement, acompressor (or de-compressor) or coder (or decoder) in the case of acompression or a coding/decoding schemes, modulator or demodulator incase of modulation, and extractor for extracting or detecting a featureor parameter such as pattern recognition or correlation analysis. Incase of filtering, passive, active or adaptive (such as Wiener orKalman) filters may be used. The conditioning circuits may apply linearor non-linear manipulations. Further, the manipulation may betime-related such as analog or digital delay-lines, integrators, orrate-based manipulation. A sensor 51 may have analog output, requiringan A/D to be connected thereto, or may have digital output. Further, theconditioning may be based on the book entitled: “Practical DesignTechniques for Sensor Signal Conditioning”, by Analog Devices, Inc.,1999 (ISBN-0-916550-20-6), which is incorporated in its entirety for allpurposes as if fully set forth herein.

Alternatively or in addition, any sensor herein, any sensor technologyherein, any sensor conditioning herein or handling circuits, or anysensor application herein, may be according to the book entitled:“Sensors and Control Systems in manufacturing”, Second Edition 2010, bySabrie Soloman, The McGraw-Hill Companies, ISBN: 978-0-07-160573-1,according to the book entitled: “Fundamentals of IndustrialInstrumentation and Process Control”, by William C. Dunn, 2005, TheMcGraw-Hill Companies, ISBN: 0-07-145735-6, or according to the bookentitled: “Sensor technology Handbook”, Edited by Jon Wilson, byNewnes-Elsevier 2005, ISBN:0-7506-7729-5, which are all incorporated intheir entirety for all purposes as if fully set forth herein. Further,the sensor 51 may be any sensor described in U.S. Patent ApplicationPublication No. 2013/0201316 to Binder et al., entitled: “System andMethod for Server Based Control”, which is incorporated in its entiretyfor all purposes as if fully set forth herein.

The sensor 51 may directly or indirectly measure the rate of change ofthe physical quantity (gradient) versus the direction around aparticular location, or between different locations. For example, atemperature gradient may describe the differences in the temperaturebetween different locations. Further, a sensor may measuretime-dependent or time-manipulated values of the phenomenon, such astime-integrated, average or Root Mean Square (RMS or rms), relating tothe square root of the mean of the squares of a series of discretevalues (or the equivalent square root of the integral in a continuouslyvarying value). Further, a parameter relating to the time dependency ofa repeating phenomenon may be measured, such as the duty-cycle, thefrequency (commonly measured in Hertz—Hz) or the period. A sensor may bebased on the Micro Electro-Mechanical Systems—MEMS (a.k.a.Micro-mechanical electrical systems) technology. A sensor may respond toenvironmental conditions such as temperature, humidity, noise,vibration, fumes, odors, toxic conditions, dust, and ventilation.

The sensor 51 may be an active sensor, requiring an external source ofexcitation. For example, resistor-based sensors such as thermistors andstrain gages are active sensors, requiring a current to pass throughthem in order to determine the resistance value, corresponding to themeasured phenomenon. Similarly, a bridge circuit based sensors areactive sensors depending or external electrical circuit for theiroperation. Alternatively or in addition, the sensor 51 may be a passivesensor, generating an electrical output without requiring any externalcircuit or any external voltage or current. Thermocouples andphotodiodes are examples or passive sensors.

The sensor 51 may measure the amount of a property or of a physicalquantity or the magnitude relating to a physical phenomenon, body orsubstance. Alternatively or in addition, the sensor 51 may be used tomeasure the time derivative thereof, such as the rate of change of theamount, the quantity or the magnitude. In the case of space relatedquantity or magnitude, the sensor 51 may measure the linear density,relating to the amount of property per length, a sensor may measure thesurface density, relating to the amount of property per area, or asensor may measure the volume density, relating to the amount ofproperty per volume. Alternatively or in addition, the sensor 51 maymeasure the amount of property per unit mass or per mole of substance.In the case of a scalar field, a sensor may further measure the quantitygradient, relating to the rate of change of property with respect toposition. Alternatively or in addition, the sensor 51 may measure theflux (or flow) of a property through a cross-section or surfaceboundary. Alternatively or in addition, a sensor may measure the fluxdensity, relating to the flow of property through a cross-section perunit of the cross-section, or through a surface boundary per unit of thesurface area. Alternatively or in addition, the sensor 51 may measurethe current, relating to the rate of flow of property through across-section or a surface boundary, or the current density, relating tothe rate of flow of property per unit through a cross-section or asurface boundary. The sensor 51 may include or consists of a transducer,defined herein as a device for converting energy from one form toanother for the purpose of measurement of a physical quantity or forinformation transfer. Further, a single sensor may be used to measuretwo or more phenomena. For example, two characteristics of the sameelement may be measured, each characteristic corresponding to adifferent phenomenon.

The sensor 51 output may have multiple states, where the sensor state isdepending upon the measured parameter of the sensed phenomenon. Thesensor 51 may be based on a two state output (such as ‘0’ or ‘1’, or‘true’ and ‘false’), such as an electric switch having two contacts,where the contacts can be in one of two states: either “closed” meaningthe contacts are touching and electricity can flow between them, or“open”, meaning the contacts are separated and the switch isnon-conducting. The sensor 51 may be a threshold switch, where theswitch changes its state upon sensing that the magnitude of the measuredparameter of a phenomenon exceeds a certain threshold. For example, thesensor 51 may be a thermostat is a temperature-operated switch used tocontrol a heating process. Another example is a voice operated switch(a.k.a. VOX), which is a switch that operates when sound over a certainthreshold is detected. It is usually used to turn on a transmitter orrecorder when someone speaks and turn it off when they stop speaking.Another example is a mercury switch (also known as a mercury tiltswitch), which is a switch whose purpose is to allow or interrupt theflow of electric current in an electrical circuit in a manner that isdependent on the switch's physical position or alignment relative to thedirection of the “pull” of earth's gravity, or other inertia. Thethreshold of a threshold based switch may be fixed or settable. Further,an actuator may be used in order to locally or remotely set thethreshold level.

In some cases, the sensor 51 operation may be based on generating astimulus or an excitation to generate influence or create a phenomenon.The entire or part of the generating or stimulating mechanism may be inthis case an integral part of the sensor, or may be regarded asindependent actuators, and thus may be controlled by the controller.Further, a sensor and an actuator, independent or integrated, may becooperatively operating as a set, for improving the sensing or theactuating functionality. For example, a light source, treated as anindependent actuator, may be used to illuminate a location, in order toallow an image sensor to faithfully and properly capture an image ofthat location. In another example, where a bridge is used to measureimpedance, the excitation voltage of the bridge may be supplied from apower supply treated and acting as an actuator.

The sensor 51 may be a piezoelectric sensor, where the piezoelectriceffect is used to measure pressure, acceleration, strain or force.Depending on how the piezoelectric material is cut, there are three mainmodes of operation: transverse longitudinal and shear. In the transverseeffect mode, a force applied along an axis generates charges in adirection perpendicular to the line of force, and in the longitudinaleffect mode, the amount of charge produced is proportional to theapplied force and is independent of size and shape of the piezoelectricelement. When using as a pressure sensor, commonly a thin membrane isused to transfer the force to the piezoelectric element, while inaccelerometer use, a mass is attached to the element, and the load ofthe mass is measured. A piezoelectric sensor element material may be apiezoelectric ceramics (such as PZT ceramic) or a single crystalmaterial. A single crystal material may be gallium phosphate, quartz,tourmaline, or Lead Magnesium Niobate-Lead Titanate (PMN-PT).

The sensor 51 may be a solid state sensor, which is typically asemiconductor device and which have no mobile parts, and commonlyenclosed as a chip. The sensor may be according to, or based on, thesensor described in U.S. Pat. No. 5,511,547 to Markle, entitled: “SolidState Sensors”, in U.S. Pat. No. 6,747,258 to Benz et al., entitled:“Intensified Hybrid Solid-State Sensor with an Insulating Layer”, inU.S. Pat. No. 5,105,087 to Jagielinski, entitled: “Large Solid StateSensor Assembly Formed from Smaller Sensors”, or in U.S. Pat. No.4,243,631 to Ryerson, entitled: “Solid State Senso”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

The sensor 51 may be a nanosensor, which is a biological, chemical orphysical sensor constructed using nanoscale components, usuallymicroscopic or submicroscopic in size. A nanosensor may be according to,or based on, the sensor described in U.S. Pat. No. 7,256,466 to Lieberet al., entitled: “Nanosensors”, in U.S. Patent Application PublicationNo. 2007/0264623 to Wang et al., entitled: “Nanosensors”, in U.S. PatentApplication Publication No. 2011/0045523 to Strano et al., entitled:“Optical Nenosensors Comprising Photoluminescent Nanostructures”, or inU.S. Patent Application Publication No. 2011/0275544 to Zhou et al.,entitled: “Microfluidic Integration with Nanosensor Platform”, which areall incorporated in their entirety for all purposes as if fully setforth herein.

The sensor 51 may include one or more sensors, each providing anelectrical output signal (such as voltage or current), or changing acharacteristic (such as resistance or impedance) in response to ameasured or detected phenomenon. The sensors may be identical, similaror different from each other, and may measure or detect the same ordifferent phenomena. Two or more sensors may be connected in series orin parallel. In the case of a changing characteristic sensor or in thecase of an active sensor, the unit may include an excitation ormeasuring circuits (such as a bridge) to generate the sensor electricalsignal. The sensor output signal may be conditioned by a signalconditioning circuit. The signal conditioner may involve time,frequency, or magnitude related manipulations. The signal conditionermay be linear or non-linear, and may include an operation or aninstrument amplifier, a multiplexer, a frequency converter, afrequency-to-voltage converter, a voltage-to-frequency converter, acurrent-to-voltage converter, a current loop converter, a chargeconverter, an attenuator, a sample-and-hold circuit, a peak-detector, avoltage or current limiter, a delay line or circuit, a level translator,a galvanic isolator, an impedance transformer, a linearization circuit,a calibrator, a passive or active (or adaptive) filter, an integrator, adeviator, an equalizer, a spectrum analyzer, a compressor or ade-compressor, a coder (or decoder), a modulator (or demodulator), apattern recognizer, a smoother, a noise remover, an average or RMScircuit, or any combination thereof. In the case of analog sensor, ananalog to digital (A/D) converter may be used to convert the conditionedsensor output signal to a digital sensor data. The unit may include acomputer for controlling and managing the unit operation, processing thedigital sensor data and handling the unit communication. The unit mayinclude a modem or transceiver coupled to a network port (such as aconnector or antenna), for interfacing and communicating over a network.

In one example, the video camera 10 is mechanically attached to thesensor 51. For example, the video camera 10 may directly mechanicallyfixed to the sensor 51. In another example, the video camera 10 may beindirectly mechanically fixed to the sensor 51, such as where they areboth attached to another structure (or frame) or component. For example,both the video camera 10 and the sensor 51 are fixedly mounted to thestructure or frame of a single enclosure, such as an enclosure 61 shownas part of an arrangement 60 in FIG. 6 or to an enclosure 62 shown aspart of an arrangement 60 b in FIG. 6b . Such mechanical attachment mayprovide a permanent relative connection, so that they both are jointlyspatially affected by any moving and positioning. In such a scenario,the sensor 51 may be a position or motion sensor, and the physicalphenomenon 54 may relate to the video camera 10 or the joint position ormotion. In such a case, a position motion detection techniques, such asegomotion, may be used as part of the VCA performed in the videoprocessor 52.

The sensor 51 may be a clinometer (a.k.a. inclinometer, tilt sensor,slope gauge, and pitch/roll indicator) for measuring angle (or slope ortilt), elevation or depression of an object, or pitch or roll (commonlywith respect to gravity), with respect to the earth ground plane, orwith respect to the horizon, commonly expressed in degrees. Theclinometers may measure inclination (positive slope), declination(negative slope), or both. A clinometer may be based on anaccelerometer, a pendulum, or on a gas bubble in liquid. Theinclinometer may be a tilt switch, such as a mercury tilt switch,commonly based on a sealed glass envelope which contains a bead ormercury. When tilted in the appropriate direction, the bead touches aset (or multiple sets) of contacts, thus completing an electricalcircuit.

The sensor 51 may be an angular rate sensor, and may be according to, orbased on, the sensor described in U.S. Pat. No. 4,759,220 to Burdess etal., entitled: “Angular Rate Sensors”, in U.S. Patent ApplicationPublication No. 2011/0041604 to Kano et al., entitled: “Angular RateSensor”, in U.S. Patent Application Publication No. 2011/0061460 toSeeger et al., entitled: “Extension-Mode Angular Velocity Sensor”, or inU.S. Patent Application Publication No. 2011/0219873 to OHTA et al.,entitled: “Angular Rate Sensor”, which are all incorporated in theirentirety for all purposes as if fully set forth herein.

The sensor 51 may be a motion sensor, and may include one or moreaccelerometers, which measures the absolute acceleration or theacceleration relative to freefall. The accelerometer may bepiezoelectric, piezoresistive, capacitive, MEMS or electromechanicalswitch accelerometer, measuring the magnitude and the direction thedevice acceleration in a single-axis, 2-axis or 3-axis(omnidirectional). Alternatively or in addition, the motion sensor maybe based on electrical tilt and vibration switch or any otherelectromechanical switch.

The sensor 51 may be a position sensor for measuring linear or angularposition (or motion). A position sensor may be an absolute positionsensor, or may be a displacement (relative or incremental) sensor,measuring a relative position, and may be an electromechanical sensor. Aposition sensor may be mechanically attached to the measured object, oralternatively may use a non-contact measurement.

A position sensor may be an angular position sensor, for measuringinvolving an angular position (or the rotation or motion) of a shaft, anaxle, or a disk. Absolute angular position sensor output indicates thecurrent position (angle) of the shaft, while incremental or displacementsensor provides information about the change, the angular speed or themotion of the shaft. An angular position sensor may be of optical type,using reflective or interruption schemes, or may be of magnetic type,such as based on variable-reluctance (VR), Eddy-current killedoscillator (ECKO), Wiegand sensing, or Hall-effect sensing, or may bebased on a rotary potentiometer. An angular position sensor may betransformer based such as a RVDT, a resolver or a synchro. An angularposition sensor may be based on an absolute or incremental rotaryencoder, and may be a mechanical or optical rotary encoder, using binaryor gray encoding schemes.

The sensor 51 may be an angular rate sensor, used to measure the angularrate, or the rotation speed, of a shaft, an axle or a disc, and may beelectromechanical (such as centrifugal switch), MEMS based, Laser based(such as Ring Laser Gyroscope—RLG), or a gyroscope (such as fiber-opticgyro) based. Some gyroscopes use the measurement of the Coriolisacceleration to determine the angular rate. An angular rate sensor maybe a tachometer, which may be based on measuring the centrifugal force,or based on optical, electric, or magnetic sensing a slotted disk.

A position sensor may be a linear position sensor, for measuring alinear displacement or position typically in a straight line, and mayuse a transformer principle such as such as LVDT, or may be based on aresistive element such as linear potentiometer. A linear position sensormay be an incremental or absolute linear encoder, and may employoptical, magnetic, capacitive, inductive, or eddy-current principles.

The sensor 51 may be a clinometer for measuring angle (such as pitch orroll) of an object, typically with respect to a plane such as the earthground plane. A clinometer may be based on an accelerometer, a pendulum,or on a gas bubble in liquid, or may be a tilt switch such as a mercurytilt switch for detecting inclination or declination with respect to adetermined tilt angle. The sensor 51 may be a gyroscope, for measuringorientation in space, such as the conventional mechanical type, a MEMSgyroscope, a piezoelectric gyroscope, a FOG, or a VSG type.

The sensor 51 may be an absolute, a relative displacement, or anincremental position sensor, and may respond to a linear or angularposition, or motion, of a sensed element. The position sensor may be anoptical type or a magnetic type angular position sensor, and may respondto an angular position or the rotation of a shaft, an axle, or a disk.The angular position sensor may be based on a variable-reluctance (VR),an Eddy-current killed oscillator (ECKO), a Wiegand sensing, or aHall-effect sensing, and may be transformer based such as an RVDT, aresolver or a synchro. The angular position sensor may be anelectromechanical type such as an absolute or an incremental, mechanicalor optical, rotary encoder. The angular position sensor may be anangular rate sensor and may respond to the angular rate, or the rotationspeed, of a shaft, an axle, or a disc, and may consist of, or comprise,a gyroscope, a tachometer, a centrifugal switch, a Ring Laser Gyroscope(RLG), or a fiber-optic gyro. The position sensor may be a linearposition sensor and may respond to a linear displacement or positionalong a line, and may consist of, or comprise, a transformer, an LVDT, alinear potentiometer, or an incremental or absolute linear encoder.

In one example, the sensor 51 is a motion sensor, and may include one ormore accelerometers, which measures the absolute acceleration or theacceleration relative to freefall. For example, one single-axisaccelerometer per axis may be used, requiring three such accelerometersfor three-axis sensing. The motion sensor may be a single or multi-axissensor, detecting the magnitude and direction of the acceleration as avector quantity, and thus can be used to sense orientation,acceleration, vibration, shock and falling. The motion sensor output maybe analog or digital signals, representing the measured values. Themotion sensor may be based on a piezoelectric accelerometer thatutilizes the piezoelectric effect of certain materials to measuredynamic changes in mechanical variables (e.g., acceleration, vibration,and mechanical shock). Piezoelectric accelerometers commonly rely onpiezoceramics (e.g., lead zirconate titanate) or single crystals (e.g.,Quartz, tourmaline). Alternatively or in addition, the motion sensor maybe based on electrical tilt and vibration switch or any otherelectromechanical switch.

The sensor 51 may be a position sensor for measuring linear or angularposition (or motion). A position sensor may be an absolute positionsensor, or may be a displacement (relative or incremental) sensor,measuring a relative position, and may further be an electromechanicalsensor. A position sensor may be mechanically attached to the measuredobject, or alternatively may use a non-contact measurement.

A position sensor may be an angular position sensor, for measuringinvolving an angular position (or the rotation or motion) of a shaft, anaxle, or a disk. Angles are commonly expressed in radians (rad), or indegrees (°), minutes (′), and seconds (″), and angular velocity commonlyuses units of radian per second (rad/s). Absolute angular positionsensor output indicates the current position (angle) of the shaft, whileincremental or displacement sensor provides information about thechange, the angular speed or the motion of the shaft. An angularposition sensor may be of optical type, using reflective or interruptionschemes. A reflective sensor is based on a light-detector that senses areflected beam from a light emitter, while an interruptive sensor isbased on interrupting the light path between the emitter and thedetector. An angular position sensor may be of magnetic type, relying ondetection based on the changes in the magnetic field. A magnetic-basedangular position sensor may be based on a variable-reluctance (VR),Eddy-Current Killed Oscillator (ECKO), Wiegand sensing, or Hall-effectsensing, used to detect a pattern in the rotating disc. A rotarypotentiometer may serve as an angular position sensor.

An angular position sensor may be based on a Rotary VariableDifferential Transformer (RVDT), used for measuring the angulardisplacement by using a type of an electrical transformer. An RVDT iscommonly composed of a salient two-pole rotor and a stator consisting ofa primary excitation coil and a pair of secondary output coils,electromagnetically coupled to the excitation coil. The coupling isproportional to the angle of the measured shaft; hence the AC outputvoltage is proportional to the angular shaft displacement. A resolverand a synchro are similar transformer based angular position sensors.

An angular position sensor may be based on a rotary encoder (a.k.a.shaft encoder), used for measuring angular position commonly by using adisc, which is rigidly fixed to the measured shaft, and containconductive, optical, or magnetic tracks. A rotary encoder may be anabsolute encoder, or may be an incremental rotary encoder, where outputis provided only when the encoder is rotating. A mechanical rotaryencoder use an insulating disc and sliding contacts, which closeelectrical circuits upon rotation of the disc. An optical rotary encoderuses a disc having transparent and opaque areas, and a light source anda photo detector to sense the optical pattern on the disc. Bothmechanical and optical rotary encoders, and may use binary or grayencoding schemes.

The sensor 51 may be an angular rate sensor, used to measure the angularrate, or the rotation speed, of a shaft, an axle or a disk. An angularrate sensor may be electromechanical, MEMS based, Laser based (such asRing Laser Gyroscope—RLG), or a gyroscope (such as fiber-optic gyro)based. Some gyroscopes use the measurement of the Coriolis accelerationto determine the angular rate.

An angular rate sensor may be a tachometer (a.k.a. RPM gauge andrevolution-counter), used to measure the rotation speed of a shaft, anaxle or a disk, commonly by units of RPM (Revolutions per Minute)annotating the number of full rotations completed in one minute aroundthe axis. A tachometer may be based on any angular position sensor, forexample sensors that are described herein, using further conditioning orprocessing to obtain the rotation speed. A tachometer may be based onmeasuring the centrifugal force, or based on sensing a slotted disk,using optical means where an optical beam is interrupted, electricalmeans where electrical contacts sense the disk, or by using magneticsensors, such as based on Hall-effect. Further, an angular rate sensormay be a centrifugal switch, which is an electric switch that operatesusing the centrifugal force created from a rotating shaft, most commonlythat of an electric motor or a gasoline engine. The switch is designedto activate or de-activate as a function of the rotational speed of theshaft.

A position sensor may be a linear position sensor, for measuring alinear displacement or position typically in a straight line. The SIunit for length is meter (m), and prefixes may be used such as nanometer(nm), micrometer, centimeter (cm), millimeter (mm), and kilometer (Km).A linear position sensor may be based on a resistance changing elementsuch as linear potentiometer.

A linear position sensor may be a Linear Variable DifferentialTransformer (LVDT) used for measuring linear displacement based on thetransformer concept. An LVDT has three coils placed in a tube, where thecenter coil serves as the primary winding coil, and the two outer coilsserve as the transformer secondary windings. The position of a slidingcylindrical ferromagnetic core is measured by changing the mutualmagnetic coupling between the windings. A linear position sensor may bea linear encoder, which may be similar to the rotary encodercounterpart, and may be based on the same principles. A linear encodermay be either incremental or absolute, and may be of optical, magnetic,capacitive, inductive, or eddy-current type. Optical linear encodertypically uses a light source such as an LED or laser diode, and mayemploy shuttering, diffraction, or holographic principles. A magneticlinear encoder may employ an active (magnetized) or passive (variablereluctance) scheme, and the position may be sensed using a sense coil,‘Hall effect’ or magneto-resistive read-head. A capacitive or inductivelinear encoder respectively measures the changes of capacitance or theinductance. Eddy-current linear encoder may be based on U.S. Pat. No.3,820,110 to Henrich et al. entitled: “Eddy Current Type Digital Encoderand Position Reference”.

The sensor 51 may consist of, or be based on, a gyroscope, for measuringorientation is space. A conventional gyroscope is a mechanical type,consisting of a wheel or disk mounted so that it can spin rapidly aboutan axis that is itself free to alter in direction. The orientation ofthe axis is not affected by tilting of the mounting; so gyroscopes arecommonly used to provide stability or maintain a reference direction innavigation systems, automatic pilots, and stabilizers. A MEMS gyroscopemay be based on vibrating element based on the Foucault pendulumconcept. A Fiber Optic Gyroscope (FOG) uses the interference or light todetect mechanical rotation. A Vibrating structure Gyroscope (VSG, a.k.a.Coriolis Vibratory Gyroscope—CVG), is based on a metal alloy resonator,and may be a piezoelectric gyroscope type where a piezoelectric materialis vibrating and the lateral motion due to centrifugal force ismeasured.

The sensor 51 may be a motion sensor, and may include one or moreaccelerometers, which measure the absolute acceleration or theacceleration relative to freefall. The accelerometer may bepiezoelectric, piezoresistive, capacitive, MEMS, or electromechanicalswitch accelerometer, measuring the magnitude and the direction thedevice acceleration in a single-axis, 2-axis or 3-axis(omnidirectional). Alternatively or in addition, the motion sensor maybe based on electrical tilt and vibration switch or any otherelectromechanical switch.

In one example, the physical phenomenon 54 comprises mechanicalvibration, defined as the measurement of a periodic process ofoscillations with respect to an equilibrium point. The oscillations maybe periodic, such as the motion of a pendulum, or random. Free vibrationoccurs when a mechanical system is set in motion with an initial inputand allowed to vibrate freely. Examples of this type of vibration arepulling a child back on a swing and letting it go, or hitting a tuningfork and letting it ring. The mechanical system vibrates at one or moreof its natural frequencies and damps down to motionlessness. Forcedvibration is when a time-varying disturbance (load, displacement orvelocity) is applied to a mechanical system. The disturbance can be aperiodic and steady-state input, a transient input, or a random input.The periodic input can be a harmonic or a non-harmonic disturbance.Examples of these types of vibration include a washing machine shakingdue to an imbalance, transportation vibration caused by an engine oruneven road, or the vibration of a building during an earthquake. Forlinear systems, the frequency of the steady-state vibration responseresulting from the application of a periodic, harmonic input is equal tothe frequency of the applied force or motion, with the responsemagnitude being dependent on the actual mechanical system. Dampedvibration: When the energy of a vibrating system is gradually dissipatedby friction and other resistances, the vibrations are said to be damped.The vibrations gradually reduce or change in frequency or intensity orcease and the system rests in its equilibrium position. An example ofthis type of vibration is the vehicular suspension dampened by the shockabsorber.

While exampled above regarding a physical phenomenon 54 that involvesorientation or motion that mechanically affects both the video camera 10and the sensor 51 as they are mechanically attached to each other, thephysical phenomenon 54 may equally comprises an environmental conditionor state that affects both the video camera 10 and the sensor 51. Theymay be jointly affected by being mechanically attached, such as directlymechanically attached to each other or being enclosed in the samehousing. Alternatively or in addition, the video camera 10 and thesensor 51 may be jointly affected by the environmental condition that ispart of the physical phenomenon 54 being in the vicinity of each other,even if they are not in the same enclosure or being mechanicallyattached to each other. For example, the distance between the videocamera 10 and the sensor 51 may be equal to, may be less than, or may behigher than, 1 cm (centimeter), 2 cm, 3 cm, 5 cm, 8 cm, 10 cm, 20 cm, 30cm, 50 cm, 80 cm, 1 m (meter), 2 m, 3 m, 5 m, 8 m, 10 m, 20 m, 30 m, 50m, 80 m, 100 m, 200 m, 300 m, 500 m, 800 m, or 1 Km (kilometer). Thesensor 51 may be operative to sense, measure, or detect theenvironmental condition, and may respond to environmental conditionssuch as temperature, humidity, noise, vibration, fumes, odors, toxicconditions, dust, and ventilation.

In one example, the physical phenomenon 54 relates to the temperature ofan object, that may be solid, liquid or gas (such as the airtemperature), in a location. A corresponding sensor 51 may be based on athermistor, which is a type of resistor whose resistance variessignificantly with temperature, and is commonly made of ceramic orpolymer material. A thermistor may be a PTC (Positive TemperatureCoefficient) type, where the resistance increases with increasingtemperatures, or may be an NTC (Negative Temperature Coefficient) type,where the resistance decreases with increasing temperatures.Alternatively (or in addition), a thermoelectric sensor may be based ona thermocouple, consisting of two different conductors (usually metalalloys), that produce a voltage proportional to a temperaturedifference. For higher accuracy and stability, an RTD (ResistanceTemperature Detector) may be used, typically consisting of a length offine wire-wound or coiled wire wrapped around a ceramic or glass core.The RTD is made of a pure material whose resistance at varioustemperatures is known (R vs. T). A common material used may be platinum,copper, or nickel. A quartz thermometer may be used as well forhigh-precision and high-accuracy temperature measurement, based on thefrequency of a quartz crystal oscillator. The temperature may bemeasured using conduction, convection, thermal radiation, or by thetransfer of energy by phase changes. The temperature may be measured indegrees Celsius (° C.) (a.k.a. Centigrade), Fahrenheit (° F.), or Kelvin(° K). In one example, the temperature sensor (or its output) is used tomeasure a temperature gradient, providing in which direction and at whatrate the temperature changes the most rapidly around a particularlocation. The temperature gradient is a dimensional quantity expressedin units of degrees (on a particular temperature scale) per unit length,such as the SI (International System of Units) unit Kelvin per meter(K/m).

In one example, the physical phenomenon 54 relates to radioactivity, anda corresponding sensor 51 may be based on a Geiger counter, measuringionizing radiation. The emission of alpha particles, beta particles orgamma rays are detected and counted by the ionization produced in alow-pressure gas ion a Geiger-Muller tube. The SI unit of radioactiveactivity is the Becquerel (Bq).

In one example, the physical phenomenon 54 relates to the ambient lightlevel, and a corresponding sensor 51 may be based on a photoelectricsensor is used to measure, sense or detect light or the luminousintensity, such as a photosensor or a photodetector. The light sensedmay be a visible light, or invisible light such as infrared,ultraviolet, X-ray or gamma rays. Such sensors may be based on thequantum mechanical effects of light on electronic materials, typicallysemiconductors such as silicon, germanium, and Indium gallium arsenide.A photoelectric sensor may be based on the photoelectric or photovoltaiceffect, such as a photodiode, phototransistor and a photomultipliertube. The photodiode typically uses a reverse biased p-n junction or PINstructure diode, and a phototransistor is in essence a bipolartransistor enclosed in a transparent case so that light can reach thebase-collector junction, and the electrons that are generated by photonsin the base-collector junction are injected into the base, and thisphotodiode current is amplified by the transistor's current gain β (orhfe). A reverse-biased LED (Light Emitting Diode) may also act as aphotodiode. Alternatively or in addition, a photosensor may be based onphotoconductivity, where the radiation or light absorption changes theconductivity of a photoconductive material, such as selenium, leadsulfide, cadmium sulfide, or polyvinylcarbazole. In such a case, thesensor may be based on photoresistor or LDR (Light Dependent Resistor),which is a resistor whose resistance decreases with increasing incidentlight intensity. In one example, Charge-Coupled Devices (CCD) and CMOS(Complementary Metal-Oxide-Semiconductor) may be used as thelight-sensitive elements, where incoming photons are converted intoelectron charges at the semiconductor-oxide interface. The sensor may bebased an Active Pixel Sensor (APS), for example as an element in animage sensor, and may be according to, or based on, the sensor describedin U.S. Pat. No. 6,549,234 to Lee, entitled: “Pixel Structure of ActivePixel Sensor (APS) with Electronic Shutter Function”, in U.S. Pat. No.6,844,897 to Andersson, entitled: “Active Pixel Sensor (APS) ReadoutStructure with Amplification”, in U.S. Pat. No. 7,342,212 to Mentzer etal., entitled: “Analog Vertical Sub-Sampling in an Active Pixel Sensor(APS) Image Sensor”, or in U.S. Pat. No. 6,476,372 to Merrill et al.,entitled: “CMOS Active Pixel Sensor Using Native Transistors”, which areall incorporated in their entirety for all purposes as if fully setforth herein.

In one example, the physical phenomenon 54 relates to an atmosphericpressure, and a corresponding sensor 51 may be based on a pressuresensor (a.k.a. pressure transducer or pressure transmitter/sender) formeasuring a pressure of gases or liquids, commonly using units of Pascal(Pa), Bar (b) (such as millibar), Atmosphere (atm), Millimeter ofMercury (mmHg), or Torr, or in terms of force per unit area such asBarye—dyne per square centimeter (Ba). Pressure sensor may indirectlymeasure other variable such as fluid/gas flow, speed, water-level, andaltitude. A pressure sensor may be a pressure switch, acting to completeor break an electric circuit in response to measured pressure magnitude.A pressure sensor may be an absolute pressure sensor, where the pressureis measured relative to a perfect vacuum, may be a gauge pressure sensorwhere the pressure is measured relative to an atmospheric pressure, maybe a vacuum pressure sensor where a pressure below atmospheric pressureis measured, may be a differential pressure sensor where the differencebetween two pressures is measured, or may be a sealed pressure sensorwhere the pressure is measured relative to some fixed pressure. Thechanges in pressure relative to altitude may serve to use a pressuresensor for altitude sensing, and the Venturi effect may be used tomeasure flow by a pressure sensor. Similarly, the depth of a submergedbody or the fluid level on contents in a tank may be measured by apressure sensor.

A pressure sensor may be of a force collector type, where a forcecollector (such a diaphragm, piston, bourdon tube, or bellows) is usedto measure strain (or deflection) due to applied force (pressure) overan area. Such sensor may be a based on the piezoelectric effect (apiezoresistive strain gauge), and may use Silicon (Monocrystalline),Polysilicon Thin Film, Bonded Metal Foil, Thick Film, or Sputtered ThinFilm. Alternatively or in addition, such force collector type sensor maybe of a capacitive type, which uses a metal, a ceramic, or a silicondiaphragm in a pressure cavity to create a variable capacitor to detectstrain due to applied pressure. Alternatively or in addition, such forcecollector type sensor may be of an electromagnetic type, where thedisplacement of a diaphragm by means of changes in inductance ismeasured. Further, in optical type the physical change of an opticalfiber, such as strain, due to applied pressure is sensed. Further, apotentiometric type may be used, where the motion of a wiper along aresistive mechanism is used to measure the strain caused by the appliedpressure. A pressure sensor may measure the stress or the changes in gasdensity, caused by the applied pressure, by using the changes inresonant frequency in a sensing mechanism, by using the changes inthermal conductivity of a gas, or by using the changes in the flow ofcharged gas particles (ions). An air pressure sensor may be a barometer,typically used to measure the atmospheric pressure, commonly used forweather forecast applications.

A pressure sensor may be according to, or based on, the sensor describedin U.S. Pat. No. 5,817,943 to Welles, II et al., entitled: “PressureSensors”, in U.S. Pat. No. 6,606,911 to Akiyama et al., entitled:“Pressure Sensors”, in U.S. Pat. No. 4,434,451 to Delatorre, entitled:“Pressure Sensors”, or in U.S. Pat. No. 5,134,887 to Bell, entitled:“Pressure Sensors”, which are all incorporated in their entirety for allpurposes as if fully set forth herein.

In one example, the physical phenomenon 54 relates to a humidity, and acorresponding sensor 51 may be based on a humidity sensor, such as ahygrometer, used for measuring the humidity in the environmental air orother gas, relating to the water vapors or the moisture content, or anywater content in a gas-vapor mixture. The hygrometer may be ahumidistat, which is a switch that responds to a relative humiditylevel, and commonly used to control humidifying or dehumidifyingequipment. The measured humidity may be an absolute humidity,corresponding to the amount of water vapor, commonly expressed in watermass per unit of volume. Alternatively or in addition, the humidity maybe relative humidity, defined as the ratio of the partial pressure ofwater vapor in an air-water mixture to the saturated vapor pressure ofwater at those conditions, commonly expressed in percent (%), or may bespecific humidity (a.k.a. humidity ratio), which is the ratio of watervapor to dry air in a particular mass. The humidity may be measured witha dew-point hygrometer, where condensation is detected by optical means.In capacitive humidity sensors, the effect of humidity on the dielectricconstant of a polymer or metal oxide material is measured. In resistivehumidity sensors, the resistance of salts or conductive polymers ismeasured. In thermal conductivity humidity sensors, the change inthermal conductivity of air due to the humidity is checked, providingindication of absolute humidity. The humidity sensor may be ahumidistat, which is a switch that responds to a relative humiditylevel, and commonly used to control humidifying or dehumidifyingequipment. The humidity sensor may be according to, or based on, thesensor described in U.S. Pat. No. 5,001,453 to Ikejiri et al., entitled:“Humidity Sensor”, in U.S. Pat. No. 6,840,103 to Lee at al., entitled:“Absolute Humidity Sensor”, in U.S. Pat. No. 6,806,722 to Shon et al.,entitled: “Polymer-Type Humidity Sensor”, or in U.S. Pat. No. 6,895,803to Seakins et al., entitled: “Humidity Sensor”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

A sensor may be an atmospheric sensor, and may be according to, or basedon, the sensor described in U.S. Patent Application Publication No.2004/0182167 to Orth et al., entitled: “Gage Pressure Output From anAbsolute Pressure Measurement Device”, in U.S. Pat. No. 4,873,481 toNelson et al., entitled: “Microwave Radiometer and Methods for SensingAtmospheric Moisture and Temperature”, in U.S. Pat. No. 3,213,010 toSaunders et al., entitled: “Vertical Drop Atmospheric Senso”, or in U.S.Pat. No. 5,604,595 to Schoen, entitled: “Long Stand-Off RangeDifferential Absorption Tomographic Atmospheric Trace Substances SensorSystems Utilizing Bistatic Configurations of Airborne and SatelliteLaser Source and Detector Reflector Platforms”, which are allincorporated in their entirety for all purposes as if fully set forthherein.

The methods and systems were exampled above regarding the physicalphenomenon 54 that affects the video camera 10 itself and sensed,measured, or detected by the sensor 51, as shown in the arrangement 50shown in FIG. 5. However, the concepts, methods and arrangements in aconfiguration where the physical phenomenon 54 affects not only thevideo camera 10 itself, but alternatively or in addition affects thescene 45 captured by the video camera 10, as illustrated pictorially bya dashed line 54 c in an arrangement 50 a shown in FIG. 5a . In such ascenario, the effect of the physical phenomenon 54 on the scene 45 iscaptured by the video camera 10 as part of the scene 45 capturing, asillustrated pictorially by a dashed line 54 d in the arrangement 50 a.The effect on the captured scene 45 is then detected and analyzed by thevideo processor 52, for providing the signal f′2(t-Δt) 55 a, asdescribed above.

For example, the physical phenomenon 54 may cause an object that is partof the captured scene 45 to move, such as a wind that causes trees orother vegetation to move. The wind speed and direction may be sensed bythe sensor 51, which may include a wind or air speed detector. Further,the motion in the captured scene 45 may be detected by the videoprocessor 52, and the wind speed and direction may be estimated forcomparison with the sensor 51 output as described herein.

In one example, all the functionalities and components shown as part ofthe arrangement 50 shown in FIG. 5 are housed in a single device orenclosure 61, as shown in an arrangement 60 shown in FIG. 6. In such aconfiguration, the camera 10, the sensor 51, the video processor 52(either as an hardware processor or as software that is executed by aprocessor), the comparator 53 (either as an hardware circuit, aprocessor, or as software that is executed by a processor), and thecombiner 41 are housed together, and may be mechanically attached, suchas being mechanically mounted to the single enclosure 61. Alternatively,the sensing related components, namely the camera 10 and the sensor 51are housed in one enclosure 62, while the processing elements,components, circuits, or functionalities, such as the combiner 41 andthe comparator 53, are housed in another distinct enclosure 63, as shownin an arrangement 60 a shown in FIG. 6a . In such a scenario, thecombination of the camera 10 and the sensor 51 may be located in onelocation or be part of a one device, while the processing elements,components, circuits, or functionalities, such as the combiner 41 andthe comparator 53, may be located in another location or be part ofanother device, that may be remotely situated from the scene 45 that isbeing captured. The video processor 52 functionality may be individuallyhoused, separately from the other enclosures. Alternatively, the videoprocessor 52 functionality may be added to the processing enclosure 63as part of a single enclosure 63 a shown as part of an arrangement 60 bshown in FIG. 6b , or may be added to the sensing enclosure 62 as partof a single enclosure 62 a shown as part of an arrangement 60 c shown inFIG. 6c . Each of the enclosures may include other elements, components,circuits, and functionalities, and may be part of a device or system.

Any enclosure herein, such as the enclosure 61 shown as part of thearrangement 60 in FIG. 6, the enclosure 62 shown as part of thearrangement 60 a in FIG. 6a , the enclosure 63 shown as part of thearrangement 60 a in FIG. 6a , the enclosure 62 shown as part of thearrangement 60 b in FIG. 6b , the enclosure 63 a shown as part of thearrangement 60 b in FIG. 6b , or the enclosure 62 a shown as part of thearrangement 60 c in FIG. 6c , and any other apparatus herein, which maybe any of the systems, devices, modules, or functionalities describedherein, may be integrated with, part of, be included in, be attached to,or be attachable to, a smartphone. The integration may be by beingenclosed in the same housing, sharing a power source (such as abattery), using the same processor, or any other integrationfunctionality. In one example, the functionality of any apparatusherein, which may be any of the systems, devices, modules, orfunctionalities described here, is used to improve, to control, orotherwise be used by the smartphone. In one example, a measured orcalculated value by any of the systems, devices, modules, orfunctionalities described herein, is output to the smartphone device orfunctionality to be used therein. Alternatively or in addition, any ofthe systems, devices, modules, or functionalities described herein isused as a sensor for the smartphone device or functionality.

Any enclosure herein, such as the enclosure 61 shown as part of thearrangement 60 in FIG. 6, the enclosure 62 shown as part of thearrangement 60 a in FIG. 6a , the enclosure 63 shown as part of thearrangement 60 a in FIG. 6a , the enclosure 62 shown as part of thearrangement 60 b in FIG. 6b , the enclosure 63 a shown as part of thearrangement 60 b in FIG. 6b , or the enclosure 62 a shown as part of thearrangement 60 c in FIG. 6c , and any other apparatus herein, which maybe any of the systems, devices, modules, or functionalities describedherein, may be structured as, may be shaped or configured to serve as,or may be integrated with, a wearable device. For example, any apparatusor device herein may be wearable on an organ such as on the person head,and the organ may be eye, ear, face, cheek, nose, mouth, lip, forehead,or chin. Alternatively or in addition, any apparatus or device hereinmay be constructed to have a form substantially similar to, may beconstructed to have a shape allowing mounting or wearing identical orsimilar to, or may be constructed to have a form to at least in partsubstitute for, headwear, eyewear, or earpiece. Any headwear herein mayconsist of, may be structured as, or may comprise, a bonnet, a headband,a cap, a crown, a fillet, a hair cover, a hat, a helmet, a hood, a mask,a turban, a veil, or a wig. Any eyewear herein may consist of, may bestructured as, or may comprise, glasses, sunglasses, a contact lens, ablindfold, or a goggle. Any earpiece herein may consist of, may bestructured as, or may comprise, a hearing aid, a headphone, a headset,or an earplug. Alternatively or in addition, any enclosure herein may bepermanently or releaseably attachable to, or may be part of, a clothingpiece of a person. The attaching may use taping, gluing, pinning,enclosing, encapsulating, a pin, or a latch and hook clip, and theclothing piece may be a top, bottom, or full-body underwear, or aheadwear, a footwear, an accessory, an outwear, a suit, a dress, askirt, or a top.

Any enclosure herein, such as the enclosure 61 shown as part of thearrangement 60 in FIG. 6, the enclosure 62 shown as part of thearrangement 60 a in FIG. 6a , the enclosure 63 shown as part of thearrangement 60 a in FIG. 6a , the enclosure 62 shown as part of thearrangement 60 b in FIG. 6b , the enclosure 63 a shown as part of thearrangement 60 b in FIG. 6b , or the enclosure 62 a shown as part of thearrangement 60 c in FIG. 6c , and any other apparatus herein, which maybe any of the systems, devices, modules, or functionalities describedherein, may be part of, comprises, or consists of, a vehicle. Anyvehicle herein may be a ground vehicle adapted to travel on land, suchas a bicycle, a car, a motorcycle, a train, an electric scooter, asubway, a train, a trolleybus, or a tram. Alternatively or in addition,the vehicle may be a buoyant or submerged watercraft adapted to travelon or in water, and the watercraft may be a ship, a boat, a hovercraft,a sailboat, a yacht, or a submarine. Alternatively or in addition, thevehicle may be an aircraft adapted to fly in air, and the aircraft maybe a fixed wing or a rotorcraft aircraft, such as an airplane, aspacecraft, a glider, a drone, or an Unmanned Aerial Vehicle (UAV). Anyvehicle herein may be a ground vehicle that may consist of, or maycomprise, an autonomous car, which may be according to levels 0, 1, 2,3, 4, or 5 of the Society of Automotive Engineers (SAE) J3016 standard.

In the exemplary 60 b shown in FIG. 6b , the system comprise twoenclosures, the enclosure 62 that includes the video camera 10 and thesensor 51, and the enclosure 63 a that includes the elements involved inanalyzing and using the data received from the sensing devices enclosure62. The two distinct devices may be spatially separated, such a beinginstalled or located in different locations. Further, one of theenclosures may be part of, or installed in, a vehicle, while the otherenclosure may be stationary.

In one exemplary arrangement 80 shown in FIG. 8, the two separateddevices communicate over a wireless network 84. The video camera 10 andthe sensor 51 are enclosed in an enclosure 86, which may correspond tothe enclosure 62, and the other functionalities are housed in anenclosure 87, which may correspond to the enclosure 63 a. A multiplexerMUX 81 may be used, in order to combine the two signals, the video datastream 43 and the sensor output signal 55 b. Such multiplexing may allowfor carrying the two signals over a single medium, such as a single datastream, and may be using Frequency Division/Domain Multiplexing (FDM) ormay be using Time Domain/Division Multiplexing (TDM). The combined MUX81 output signal is then fed to a wireless transceiver 82 a, thatconverts the signal into a wireless signal that can be transmitted tothe air via an antenna 83 a. The wirelessly transmitted signal iscarried over the wireless network 84, and received by a mating wirelesstransceiver 82 b via a mating antenna 83 b. The received signal is thende-multiplexed by de-multiplexer DEMUX 85, which mates with the MUX 81,and operates to separate the combined signals to the original ones.Specifically, the combined received signal is split by the DEMUX 85 to asignal 43 a that represents the transmitted video data stream 43, and ispreferably identical to it, and to a signal 55 c that represents thesensor output 55 b, and is preferably identical to it. The operation ofthe receiving device 87 is then identical or similar to the operation ofthe device 63 a, as described herein.

In one example, the wireless network 84 may be using, may be accordingto, may be compatible with, or may be based on, an Near FieldCommunication (NFC) using passive or active communication mode, may usethe 13.56 MHz frequency band, data rate may be 106 Kb/s, 212 Kb/s, or424 Kb/s, the modulation may be Amplitude-Shift-Keying (ASK), and mayfurther be according to, compatible with, or based on, ISO/IEC 18092,ECMA-340, ISO/IEC 21481, or ECMA-352. In this scenario, each of thewireless transceivers 82 a and 82 b may be an NFC modem or transceiver,and each of the antennas 83 a and 83 b may be an NFC antenna.Alternatively or in addition, the wireless network 84 may be using, maybe according to, may be compatible with, or may be based on, a PersonalArea Network (PAN) that may be according to, or based on, Bluetooth™ orIEEE 802.15.1-2005 standards that may be, each of the wirelesstransceivers 82 a and 82 b may be a PAN modem, and each of the antennas83 a and 83 b may be a PAN antenna. In one example, the Bluetooth is aBluetooth Low-Energy (BLE) standard. Further, the PAN may be a wirelesscontrol network according to, or based on, Zigbee™ or Z-Wave™ standards,such as IEEE 802.15.4-2003. Alternatively or in addition, the wirelessnetwork 84 may be using, may be according to, may be compatible with, ormay be based on, an analog Frequency Modulation (FM) over license-freeband such as the LPD433 standard that uses frequencies with the ITUregion 1 ISM band of 433.050 MHz to 434.790 MHz, each of the wirelesstransceivers 82 a and 82 b may be an LPD433 modem, and each of theantennas 83 a and 83 b may be an LPD433 antenna.

Alternatively or in addition, the wireless network 84 may be using, maybe according to, may be compatible with, or may be based on, a WirelessLocal Area Network (WLAN) that may be according to, or based on, IEEE802.11a, IEEE 802.11b, IEEE 802.11g, IEEE 802.11n, or IEEE 802.11acstandards, each of the wireless transceivers 82 a and 82 b may be a WLANmodem, and each of the antennas 83 a and 83 b may be a WLAN antenna.

Alternatively or in addition, the wireless network 84 may be using, maybe according to, may be compatible with, or may be based on, a wirelessbroadband network or a Wireless Wide Area Network (WWAN), each of thewireless transceivers 82 a and 82 b may be a WWAN modem, and each of theantennas 83 a and 83 b may be a WWAN antenna. The WWAN may be a WiMAXnetwork such as according to, or based on, IEEE 802.16-2009, each of thewireless transceivers 82 a and 82 b may be a WiMAX modem, and each ofthe antennas 83 a and 83 b may be a WiMAX antenna. Alternatively or inaddition, the WWAN may be a cellular telephone network, each of thewireless transceivers 82 a and 82 b may be a cellular modem, and each ofthe antennas 83 a and 83 b may be a cellular antenna. The WWAN may be aThird Generation (3G) network and may use UMTS W-CDMA, UMTS HSPA, UMTSTDD, CDMA2000 1×RTT, CDMA2000 EV-DO, or GSM EDGE-Evolution. The cellulartelephone network may be a Fourth Generation (4G) network and may useHSPA+, Mobile WiMAX, LTE, LTE-Advanced, MBWA, or may be based on IEEE802.20-2008. Alternatively or in addition, the wireless network 84 maybe using, may be using licensed or an unlicensed radio frequency band,such as the Industrial, Scientific and Medical (ISM) radio band.

Alternatively or in addition, the wireless network 84 may use aDedicated Short-Range Communication (DSRC), that may be according to,compatible with, or based on, European Committee for Standardization(CEN) EN 12253:2004, EN 12795:2002, EN 12834:2002, EN 13372:2004, or ENISO 14906:2004 standard, or may be according to, compatible with, orbased on, IEEE 802.11p, IEEE 1609.1-2006, IEEE 1609.2, IEEE 1609.3, IEEE1609.4, or IEEE1609.5.

Any arrangement or method described herein may be used as part of aVirtual Reality (VR) system. For example, the output 44 of the combiner41 may be fed to an HMD, allowing a person wearing the HMD to watch thecaptured scene 45, with the additional data 42 synchronously overlayedover the video data captured by the video camera 10.

While exampled above regarding an optical-based imaging video camera 10that is operative to capture images or scenes in a visible ornon-visible spectrum, any method or system herein may equally use aLiDAR camera or scanner, as well as thermal camera, as a substitute tothe video camera 10.

While exampled herein, such as in the arrangement 60 shown in FIG. 6,that the Δt 56 is a positive value, corresponding to that the videostream signal f1(t-Δ′t) 43 is lagging after the sensor 51 output signalf2(t) 55 b, any method and apparatus herein equally applies to ascenario where the video stream signal f1(t-Δ′t) 43 is advanced, and thesensor 51 output signal f2(t) 55 b is lagging beyond the video streamsignal f1(t-Δ′t) 43. Mathematically and practically, such a scenario isequivalent to a negative value of the time delay Δt 56. The absolutevalue of such a negative time delay may be equal to, higher than, orless than, 1 millisecond, 2 milliseconds, 5 milliseconds, 10milliseconds, 20 milliseconds, 30 milliseconds, 50 milliseconds, 100milliseconds, 200 milliseconds, 500 milliseconds, 1 second, 2 seconds, 5seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2, minutes, 5minutes, 10 minutes, 20 minutes, 30 minutes, 1 hour, 2 hours, 5 hours,or 10 hours.

Any apparatus herein, which may be any of the systems, devices, modules,or functionalities described herein, may be integrated with asmartphone. The integration may be by being enclosed in the samehousing, sharing a power source (such as a battery), using the sameprocessor, or any other integration functionality. In one example, thefunctionality of any apparatus herein, which may be any of the systems,devices, modules, or functionalities described here, is used to improve,to control, or otherwise be used by the smartphone. In one example, ameasured or calculated value by any of the systems, devices, modules, orfunctionalities described herein, is output to the smartphone device orfunctionality to be used therein. Alternatively or in addition, any ofthe systems, devices, modules, or functionalities described herein isused as a sensor for the smartphone device or functionality.

Any part of, or the whole of, any of the methods described herein may beprovided as part of, or used as, an Application Programming Interface(API), defined as an intermediary software serving as the interfaceallowing the interaction and data sharing between an applicationsoftware and the application platform, across which few or all servicesare provided, and commonly used to expose or use a specific softwarefunctionality, while protecting the rest of the application. The API maybe based on, or according to, Portable Operating System Interface(POSIX) standard, defining the API along with command line shells andutility interfaces for software compatibility with variants of Unix andother operating systems, such as POSIX.1-2008 that is simultaneouslyIEEE STD. 1003.1™-2008 entitled: “Standard for InformationTechnology—Portable Operating System Interface (POSIX(R)) Description”,and The Open Group Technical Standard Base Specifications, Issue 7, IEEESTD. 1003.1™, 2013 Edition.

Any part of, or whole of, any of the methods described herein may beimplemented by a processor, or by a processor that is part of a devicethat in integrated with a digital camera, and may further be used inconjunction with various devices and systems, for example a device maybe a Personal Computer (PC), a desktop computer, a mobile computer, alaptop computer, a notebook computer, a tablet computer, a servercomputer, a handheld computer, a handheld device, a Personal DigitalAssistant (PDA) device, a cellular handset, a handheld PDA device, anon-board device, an off-board device, a hybrid device, a vehiculardevice, a non-vehicular device, a mobile or portable device, or anon-mobile or non-portable device.

Any device herein may serve as a client device in the meaning ofclient/server architecture, commonly initiating requests for receivingservices, functionalities, and resources, from other devices (servers orclients). Each of the these devices may further employ, store,integrate, or operate a client-oriented (or end-point dedicated)operating system, such as Microsoft Windows® (including the variants:Windows 7, Windows XP, Windows 8, and Windows 8.1, available fromMicrosoft Corporation, headquartered in Redmond, Wash., U.S.A.), Linux,and Google Chrome OS available from Google Inc. headquartered inMountain View, Calif., U.S.A. Further, each of the these devices mayfurther employ, store, integrate, or operate a mobile operating systemsuch as Android (available from Google Inc. and includes variants suchas version 2.2 (Froyo), version 2.3 (Gingerbread), version 4.0 (IceCream Sandwich), Version 4.2 (Jelly Bean), and version 4.4 (KitKat)),iOS (available from Apple Inc., and includes variants such as versions3-7), Windows® Phone (available from Microsoft Corporation and includesvariants such as version 7, version 8, or version 9), or Blackberry®operating system (available from BlackBerry Ltd., headquartered inWaterloo, Ontario, Canada). Alternatively or in addition, each of thedevices that are not denoted herein as servers may equally function as aserver in the meaning of client/server architecture. Any one of theservers herein may be a web server using Hyper Text Transfer Protocol(HTTP) that responds to HTTP requests via the Internet, and any requestherein may be an HTTP request.

Examples of web browsers include Microsoft Internet Explorer (availablefrom Microsoft Corporation, headquartered in Redmond, Wash., U.S.A.),Google Chrome which is a freeware web browser (developed by Google,headquartered in Googleplex, Mountain View, Calif., U.S.A.), Opera™(developed by Opera Software ASA, headquartered in Oslo, Norway), andMozilla Firefox® (developed by Mozilla Corporation headquartered inMountain View, Calif., U.S.A.). The web-browser may be a mobile browser,such as Safari (developed by Apple Inc. headquartered in Apple Campus,Cupertino, Calif., U.S.A), Opera Mini™ (developed by Opera Software ASA,headquartered in Oslo, Norway), and Android web browser.

Any device herein may be integrated with part or an entire appliance.The appliance primary function may be associated with food storage,handling, or preparation, such as microwave oven, an electric mixer, astove, an oven, or an induction cooker for heating food, or theappliance may be a refrigerator, a freezer, a food processor, adishwashers, a food blender, a beverage maker, a coffeemaker, or aniced-tea maker. The appliance primary function may be associated withenvironmental control such as temperature control, and the appliance mayconsist of, or may be part of, an HVAC system, an air conditioner or aheater. The appliance primary function may be associated with cleaning,such as a washing machine, a clothes dryer for cleaning clothes, or avacuum cleaner. The appliance primary function may be associated withwater control or water heating. The appliance may be an answeringmachine, a telephone set, a home cinema system, a HiFi system, a CD orDVD player, an electric furnace, a trash compactor, a smoke detector, alight fixture, or a dehumidifier. The appliance may be a handheldcomputing device or a battery-operated portable electronic device, suchas a notebook or laptop computer, a media player, a cellular phone, aPersonal Digital Assistant (PDA), an image processing device, a digitalcamera, or a video recorder. The integration with the appliance mayinvolve sharing a component such as housing in the same enclosure,sharing the same connector such as sharing a power connector forconnecting to a power source, where the integration involves sharing thesame connector for being powered from the same power source. Theintegration with the appliance may involve sharing the same powersupply, sharing the same processor, or mounting onto the same surface.

Any steps described herein may be sequential, and performed in thedescribed order. For example, in a case where a step is performed inresponse to another step, or upon completion of another step, the stepsare executed one after the other. However, in case where two or moresteps are not explicitly described as being sequentially executed, thesesteps may be executed in any order or may be simultaneously performed.Two or more steps may be executed by two different network elements, orin the same network element, and may be executed in parallel usingmultiprocessing or multitasking.

A ‘nominal’ value herein refers to a designed, expected, or targetvalue. In practice, a real or actual value is used, obtained, or exists,which varies within a tolerance from the nominal value, typicallywithout significantly affecting functioning. Common tolerances are 20%,15%, 10%, 5%, or 1% around the nominal value.

Discussions herein utilizing terms such as, for example, “processing,”“computing,” “calculating,” “determining,” “establishing”, “analyzing”,“checking”, or the like, may refer to operation(s) and/or process(es) ofa computer, a computing platform, a computing system, or otherelectronic computing device, that manipulate and/or transform datarepresented as physical (e.g., electronic) quantities within thecomputer's registers and/or memories into other data similarlyrepresented as physical quantities within the computer's registersand/or memories or other information storage medium that may storeinstructions to perform operations and/or processes.

Throughout the description and claims of this specification, the word“couple”, and variations of that word such as “coupling”, “coupled”, and“couplable”, refers to an electrical connection (such as a copper wireor soldered connection), a logical connection (such as through logicaldevices of a semiconductor device), a virtual connection (such asthrough randomly assigned memory locations of a memory device) or anyother suitable direct or indirect connections (including combination orseries of connections), for example for allowing the transfer of power,signal, or data, as well as connections formed through interveningdevices or elements.

The arrangements and methods described herein may be implemented usinghardware, software or a combination of both. The term “integration” or“software integration” or any other reference to the integration of twoprograms or processes herein refers to software components (e.g.,programs, modules, functions, processes etc.) that are (directly or viaanother component) combined, working or functioning together or form awhole, commonly for sharing a common purpose or a set of objectives.Such software integration can take the form of sharing the same programcode, exchanging data, being managed by the same manager program,executed by the same processor, stored on the same medium, sharing thesame GUI or other user interface, sharing peripheral hardware (such as amonitor, printer, keyboard and memory), sharing data or a database, orbeing part of a single package. The term “integration” or “hardwareintegration” or integration of hardware components herein refers tohardware components that are (directly or via another component)combined, working or functioning together or form a whole, commonly forsharing a common purpose or set of objectives. Such hardware integrationcan take the form of sharing the same power source (or power supply) orsharing other resources, exchanging data or control (e.g., bycommunicating), being managed by the same manager, physically connectedor attached, sharing peripheral hardware connection (such as a monitor,printer, keyboard and memory), being part of a single package or mountedin a single enclosure (or any other physical collocating), sharing acommunication port, or used or controlled with the same software orhardware. The term “integration” herein refers (as applicable) to asoftware integration, a hardware integration, or any combinationthereof.

The term “port” refers to a place of access to a device, electricalcircuit or network, where energy or signal may be supplied or withdrawn.The term “interface” of a networked device refers to a physicalinterface, a logical interface (e.g., a portion of a physical interfaceor sometimes referred to in the industry as a sub-interface—for example,such as, but not limited to a particular VLAN associated with a networkinterface), and/or a virtual interface (e.g., traffic grouped togetherbased on some characteristic—for example, such as, but not limited to, atunnel interface). As used herein, the term “independent” relating totwo (or more) elements, processes, or functionalities, refers to ascenario where one does not affect nor preclude the other. For example,independent communication such as over a pair of independent data routesmeans that communication over one data route does not affect norpreclude the communication over the other data routes.

The term “processor” is meant to include any integrated circuit or otherelectronic device (or collection of devices) capable of performing anoperation on at least one instruction including, without limitation,Reduced Instruction Set Core (RISC) processors, CISC microprocessors,Microcontroller Units (MCUs), CISC-based Central Processing Units(CPUs), and Digital Signal Processors (DSPs). The hardware of suchdevices may be integrated onto a single substrate (e.g., silicon “die”),or distributed among two or more substrates. Furthermore, variousfunctional aspects of the processor may be implemented solely assoftware or firmware associated with the processor.

A non-limiting example of a processor may be 80186 or 80188 availablefrom Intel Corporation located at Santa-Clara, Calif., USA. The 80186and its detailed memory connections are described in the manual“80186/80188 High-Integration 16-Bit Microprocessors” by IntelCorporation, which is incorporated in its entirety for all purposes asif fully set forth herein. Other non-limiting example of a processor maybe MC68360 available from Motorola Inc. located at Schaumburg, Ill.,USA. The MC68360 and its detailed memory connections are described inthe manual “MC68360 Quad Integrated Communications Controller—User'sManual” by Motorola, Inc., which is incorporated in its entirety for allpurposes as if fully set forth herein. While exampled above regarding anaddress bus having an 8-bit width, other widths of address buses arecommonly used, such as the 16-bit, 32-bit and 64-bit. Similarly, whileexampled above regarding a data bus having an 8-bit width, other widthsof data buses are commonly used, such as 16-bit, 32-bit and 64-bitwidth. In one example, the processor consists of, comprises, or is partof, Tiva™ TM4C123GH6PM Microcontroller available from Texas InstrumentsIncorporated (Headquartered in Dallas, Tex., U.S.A.), described in adata sheet published 2015 by Texas Instruments Incorporated[DS-TM4C123GH6PM-15842.2741, SPMS376E, Revision 15842.2741 June 2014],entitled: “Tiva™ TM4C123GH6PM Microcontroller—Data Sheet”, which isincorporated in its entirety for all purposes as if fully set forthherein, and is part of Texas Instrument's Tiva™ C Seriesmicrocontrollers family that provide designers a high-performance ARM®Cortex™-M-based architecture with a broad set of integrationcapabilities and a strong ecosystem of software and development tools.Targeting performance and flexibility, the Tiva™ C Series architectureoffers an 80 MHz Cortex-M with FPU, a variety of integrated memories andmultiple programmable GPIO. Tiva™ C Series devices offer consumerscompelling cost-effective solutions by integrating application-specificperipherals and providing a comprehensive library of software toolswhich minimize board costs and design-cycle time. Offering quickertime-to-market and cost savings, the Tiva™ C Series microcontrollers arethe leading choice in high-performance 32-bit applications. Targetingperformance and flexibility, the Tiva™ C Series architecture offers an80 MHz Cortex-M with FPU, a variety of integrated memories and multipleprogrammable GPIO. Tiva™ C Series devices offer consumers compellingcost-effective solutions.

The terms “memory” and “storage” are used interchangeably herein andrefer to any physical component that can retain or store information(that can be later retrieved) such as digital data on a temporary orpermanent basis, typically for use in a computer or other digitalelectronic device. A memory can store computer programs or any othersequence of computer readable instructions, or data, such as files,text, numbers, audio and video, as well as any other form of informationrepresented as a string or structure of bits or bytes. The physicalmeans of storing information may be electrostatic, ferroelectric,magnetic, acoustic, optical, chemical, electronic, electrical, ormechanical. A memory may be in a form of an Integrated Circuit (IC,a.k.a. chip or microchip). Alternatively or in addition, a memory may bein the form of a packaged functional assembly of electronic components(module). Such module may be based on a Printed Circuit Board (PCB) suchas PC Card according to Personal Computer Memory Card InternationalAssociation (PCMCIA) PCMCIA 2.0 standard, or a Single In-line MemoryModule (SIMM) or a Dual In-line Memory Module (DIMM), standardized underthe JEDEC JESD-21C standard. Further, a memory may be in the form of aseparately rigidly enclosed box such as an external Hard-Disk Drive(HDD). Capacity of a memory is commonly featured in bytes (B), where theprefix ‘K’ is used to denote kilo=2¹⁰=1024⁴=1024, the prefix ‘M’ is usedto denote mega=2²⁰=1024²=1,048,576, the prefix ‘G’ is used to denoteGiga=2³⁰=1024³=1,073,741,824, and the prefix ‘T’ is used to denotetera=2⁴⁰=1024⁴=1,099,511,627,776.

As used herein, the term “Integrated Circuit” (IC) shall include anytype of integrated device of any function where the electronic circuitis manufactured by the patterned diffusion of trace elements into thesurface of a thin substrate of semiconductor material (e.g., Silicon),whether single or multiple die, or small or large scale of integration,and irrespective of process or base materials (including, withoutlimitation Si, SiGe, CMOS and GAs) including, without limitation,applications specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), digital processors (e.g., DSPs, CISCmicroprocessors, or RISC processors), so-called “system-on-a-chip” (SoC)devices, memory (e.g., DRAM, SRAM, flash memory, ROM), mixed-signaldevices, and analog ICs.

The circuits in an IC are typically contained in a silicon piece or in asemiconductor wafer, and commonly packaged as a unit. The solid-statecircuits commonly include interconnected active and passive devices,diffused into a single silicon chip. Integrated circuits can beclassified into analog, digital and mixed signal (both analog anddigital on the same chip). Digital integrated circuits commonly containmany of logic gates, flip-flops, multiplexers, and other circuits in afew square millimeters. The small size of these circuits allows highspeed, low power dissipation, and reduced manufacturing cost comparedwith board-level integration. Further, a multi-chip module (MCM) may beused, where multiple integrated circuits (ICs), the semiconductor dies,or other discrete components are packaged onto a unifying substrate,facilitating their use as a single component (as though a larger IC).

The term “computer-readable medium” (or “machine-readable medium”) asused herein is an extensible term that refers to any medium or anymemory, that participates in providing instructions to a processor forexecution, or any mechanism for storing or transmitting information in aform readable by a machine (e.g., a computer). Such a medium may storecomputer-executable instructions to be executed by a processing elementand/or software, and data that is manipulated by a processing elementand/or software, and may take many forms, including but not limited to,non-volatile medium, volatile medium, and transmission medium.Transmission media includes coaxial cables, copper wire and fiberoptics. Transmission media can also take the form of acoustic or lightwaves, such as those generated during radio-wave and infrared datacommunications, or other form of propagating signals (e.g., carrierwaves, infrared signals, digital signals, etc.). Common forms ofcomputer-readable media include, for example, a floppy disk, a flexibledisk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM,any other optical medium, punch-cards, paper-tape, any other physicalmedium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave as describedhereinafter, or any other medium from which a computer can read.

The term “computer” is used generically herein to describe any number ofcomputers, including, but not limited to personal computers, embeddedprocessing elements and systems, software, ASICs, chips, workstations,mainframes, etc. Any computer herein may consist of, or be part of, ahandheld computer, including any portable computer that is small enoughto be held and operated while holding in one hand or fit into a pocket.Such a device, also referred to as a mobile device, typically has adisplay screen with touch input and/or miniature keyboard. Non-limitingexamples of such devices include Digital Still Camera (DSC), Digitalvideo Camera (DVC or digital camcorder), Personal Digital Assistant(PDA), and mobile phones and Smartphones. The mobile devices may combinevideo, audio and advanced communication capabilities, such as PAN andWLAN. A mobile phone (also known as a cellular phone, cell phone and ahand phone) is a device which can make and receive telephone calls overa radio link whilst moving around a wide geographic area, by connectingto a cellular network provided by a mobile network operator. The callsare to and from the public telephone network, which includes othermobiles and fixed-line phones across the world. The Smartphones maycombine the functions of a personal digital assistant (PDA), and mayserve as portable media players and camera phones with high-resolutiontouch-screens, web browsers that can access, and properly display,standard web pages rather than just mobile-optimized sites, GPSnavigation, Wi-Fi and mobile broadband access. In addition to telephony,the Smartphones may support a wide variety of other services such astext messaging, MMS, email, Internet access, short-range wirelesscommunications (infrared, Bluetooth), business applications, gaming andphotography.

Some embodiments may be used in conjunction with various devices andsystems, for example, a Personal Computer (PC), a desktop computer, amobile computer, a laptop computer, a notebook computer, a tabletcomputer, a server computer, a handheld computer, a handheld device, aPersonal Digital Assistant (PDA) device, a cellular handset, a handheldPDA device, an on-board device, an off-board device, a hybrid device, avehicular device, a non-vehicular device, a mobile or portable device, anon-mobile or non-portable device, a wireless communication station, awireless communication device, a wireless Access Point (AP), a wired orwireless router, a wired or wireless modem, a wired or wireless network,a Local Area Network (LAN), a Wireless LAN (WLAN), a Metropolitan AreaNetwork (MAN), a Wireless MAN (WMAN), a Wide Area Network (WAN), aWireless WAN (WWAN), a Personal Area Network (PAN), a Wireless PAN(WPAN), devices and/or networks operating substantially in accordancewith existing IEEE 802.11, 802.11a, 802.11b, 802.11g, 802.11k, 802.11n,802.11r, 802.16, 802.16d, 802.16e, 802.20, 802.21 standards and/orfuture versions and/or derivatives of the above standards, units and/ordevices which are part of the above networks, one way and/or two-wayradio communication systems, cellular radio-telephone communicationsystems, a cellular telephone, a wireless telephone, a PersonalCommunication Systems (PCS) device, a PDA device which incorporates awireless communication device, a mobile or portable Global PositioningSystem (GPS) device, a device which incorporates a GPS receiver ortransceiver or chip, a device which incorporates an RFID element orchip, a Multiple Input Multiple Output (AMMO) transceiver or device, aSingle Input Multiple Output (SIMO) transceiver or device, a MultipleInput Single Output (MISO) transceiver or device, a device having one ormore internal antennas and/or external antennas, Digital Video Broadcast(DVB) devices or systems, multi-standard radio devices or systems, awired or wireless handheld device (e.g., BlackBerry, Palm Treo), aWireless Application Protocol (WAP) device, or the like.

As used herein, the terms “program”, “programmable”, and “computerprogram” are meant to include any sequence or human or machinecognizable steps, which perform a function. Such programs are notinherently related to any particular computer or other apparatus, andmay be rendered in virtually any programming language or environment,including, for example, C/C++, Fortran, COBOL, PASCAL, assemblylanguage, markup languages (e.g., HTML, SGML, XML, VoXML), and thelikes, as well as object-oriented environments such as the Common ObjectRequest Broker Architecture (CORBA), Java™ (including J2ME, Java Beans,etc.) and the like, as well as in firmware or other implementations.Generally, program modules include routines, programs, objects,components, data structures, etc., that perform particular tasks orimplement particular abstract data types.

The terms “task” and “process” are used generically herein to describeany type of running programs, including, but not limited to a computerprocess, task, thread, executing application, operating system, userprocess, device driver, native code, machine or other language, etc.,and can be interactive and/or non-interactive, executing locally and/orremotely, executing in foreground and/or background, executing in theuser and/or operating system address spaces, a routine of a libraryand/or standalone application, and is not limited to any particularmemory partitioning technique. The steps, connections, and processing ofsignals and information illustrated in the figures, including, but notlimited to, any block and flow diagrams and message sequence charts, maytypically be performed in the same or in a different serial or parallelordering and/or by different components and/or processes, threads, etc.,and/or over different connections and be combined with other functionsin other embodiments, unless this disables the embodiment or a sequenceis explicitly or implicitly required (e.g., for a sequence of readingthe value, processing the value: the value must be obtained prior toprocessing it, although some of the associated processing may beperformed prior to, concurrently with, and/or after the read operation).Where certain process steps are described in a particular order or wherealphabetic and/or alphanumeric labels are used to identify certainsteps, the embodiments of the invention are not limited to anyparticular order of carrying out such steps. In particular, the labelsare used merely for convenient identification of steps, and are notintended to imply, specify or require a particular order for carryingout such steps. Furthermore, other embodiments may use more or lesssteps than those discussed herein. The invention may also be practicedin distributed computing environments where tasks are performed byremote processing devices that are linked through a communicationsnetwork. In a distributed computing environment, program modules may belocated in both local and remote memory storage devices.

Operating system. An Operating System (OS) is software that managescomputer hardware resources and provides common services for computerprograms. The operating system is an essential component of any systemsoftware in a computer system, and most application programs usuallyrequire an operating system to function. For hardware functions such asinput/output and memory allocation, the operating system acts as anintermediary between programs and the computer hardware, although theapplication code is usually executed directly by the hardware and willfrequently make a system call to an OS function or be interrupted by it.Common features typically supported by operating systems include processmanagement, interrupts handling, memory management, file system, devicedrivers, networking (such as TCP/IP and UDP), and Input/Output (I/O)handling. Examples of popular modern operating systems include Android,BSD, iOS, Linux, OS X, QNX, Microsoft Windows, Windows Phone, and IBMz/OS.

Any software or firmware herein may comprise an operating system thatmay be a mobile operating system. The mobile operating system mayconsist of, may comprise, may be according to, or may be based on,Android version 2.2 (Froyo), Android version 2.3 (Gingerbread), Androidversion 4.0 (Ice Cream Sandwich), Android Version 4.2 (Jelly Bean),Android version 4.4 (KitKat)), Apple iOS version 3, Apple iOS version 4,Apple iOS version 5, Apple iOS version 6, Apple iOS version 7, MicrosoftWindows® Phone version 7, Microsoft Windows® Phone version 8, MicrosoftWindows® Phone version 9, or Blackberry® operating system. Any OperatingSystem (OS) herein, such as any server or client operating system, mayconsists of, include, or be based on a real-time operating system(RTOS), such as FreeRTOS, SafeRTOS, QNX, VxWorks, or Micro-ControllerOperating Systems (μC/OS).

Any apparatus herein, may be a client device that may typically functionas a client in the meaning of client/server architecture, commonlyinitiating requests for receiving services, functionalities, andresources, from other devices (servers or clients). Each of the thesedevices may further employ, store, integrate, or operate aclient-oriented (or end-point dedicated) operating system, such asMicrosoft Windows® (including the variants: Windows 7, Windows XP,Windows 8, and Windows 8.1, available from Microsoft Corporation,headquartered in Redmond, Wash., U.S.A.), Linux, and Google Chrome OSavailable from Google Inc. headquartered in Mountain View, Calif.,U.S.A. Further, each of the these devices may further employ, store,integrate, or operate a mobile operating system such as Android(available from Google Inc. and includes variants such as version 2.2(Froyo), version 2.3 (Gingerbread), version 4.0 (Ice Cream Sandwich),Version 4.2 (Jelly Bean), and version 4.4 (KitKat), iOS (available fromApple Inc., and includes variants such as versions 3-7), Windows® Phone(available from Microsoft Corporation and includes variants such asversion 7, version 8, or version 9), or Blackberry® operating system(available from BlackBerry Ltd., headquartered in Waterloo, Ontario,Canada). Alternatively or in addition, each of the devices that are notdenoted herein as a server, may equally function as a server in themeaning of client/server architecture. Any Operating System (OS) herein,such as any server or client operating system, may consists of, include,or be based on a real-time operating system (RTOS), such as FreeRTOS,SafeRTOS, QNX, VxWorks, or Micro-Controller Operating Systems (μC/OS).

Any sensor herein, such as the sensor 51, may be a piezoelectric sensor,where the piezoelectric effect is used to measure pressure,acceleration, strain or force, and may use transverse, longitudinal, orshear effect mode. A thin membrane may be used to transfer and measurepressure, while mass may be used for acceleration measurement. Apiezoelectric sensor element material may be a piezoelectric ceramics(such as PZT ceramic) or a single crystal material. A single crystalmaterial may be gallium phosphate, quartz, tourmaline, or Lead MagnesiumNiobate-Lead Titanate (PMN-PT). Any sensor herein, such as the sensor51, may be a motion sensor, and may include one or more accelerometers,which measure the absolute acceleration or the acceleration relative tofreefall. The accelerometer may be piezoelectric, piezoresistive,capacitive, MEMS, or electromechanical switch accelerometer, measuringthe magnitude and the direction the device acceleration in asingle-axis, 2-axis or 3-axis (omnidirectional). Alternatively or inaddition, the motion sensor may be based on electrical tilt andvibration switch or any other electromechanical switch.

The corresponding structures, materials, acts, and equivalents of allmeans plus function elements in the claims below are intended to includeany structure, or material, for performing the function in combinationwith other claimed elements as specifically claimed. The description ofthe present invention has been presented for purposes of illustrationand description, but is not intended to be exhaustive or limited to theinvention in the form disclosed. The present invention should not beconsidered limited to the particular embodiments described above, butrather should be understood to cover all aspects of the invention asfairly set out in the attached claims. Various modifications, equivalentprocesses, as well as numerous structures to which the present inventionmay be applicable, will be readily apparent to those skilled in the artto which the present invention is directed upon review of the presentdisclosure.

All publications, standards, patents, and patent applications cited inthis specification are incorporated herein by reference as if eachindividual publication, patent, or patent application were specificallyand individually indicated to be incorporated by reference and set forthin its entirety herein.

The invention claimed is:
 1. A method for estimating a delay of a videodata stream from a Digital Video Camera (DVC), for use with a physicalphenomenon that affects the video camera or a scene captured by thevideo camera, the method comprising: receiving, from the digital videocamera, the video data stream; producing, by a video processor, a firstsignal that estimates the physical phenomenon value, by processing thevideo data stream for detecting the effect of the physical phenomenon onthe captured video; receiving, from a sensor, a second signal that isresponsive to the physical phenomenon value; estimating a positive ornegative time delay value between the first and second signals bycomparing therebetween; and combining the video data stream withadditional data by synchronizing using the estimated time delay value.2. The method according to claim 1, wherein the time delay value isestimated in response to an event, and the time delay value continuouslyused for the combining.
 3. The method according to claim 2, wherein thetime delay value is estimated in response to a user control.
 4. Themethod according to claim 2, wherein the time delay value is estimatedin response to a power-up process.
 5. The method according to claim 1,wherein the time delay value is continuously estimated and used for thecombining.
 6. The method according to claim 1, wherein the time delayvalue is periodically estimated.
 7. The method according to claim 6,wherein the time delay value is periodically estimated every at least 1second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1hour, 2 hours, 5 hours, 10 hours, 1 day, 2 days, 4 days, 1 week, 2weeks, 3 weeks, or 1 months.
 8. The method according to claim 6, whereinthe time delay value is periodically estimated every no more than 1second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1hour, 2 hours, 5 hours, 10 hours, 1 day, 2 days, 4 days, 1 week, 2weeks, 3 weeks, or 1 months.
 9. The method according to claim 1, whereinthe comparing comprises comparing of the first and second signals duringa time-interval.
 10. The method according to claim 9, wherein thetime-interval is less than 1 millisecond, 2 milliseconds, 5milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 50milliseconds, 100 milliseconds, 200 milliseconds, 500 milliseconds, 1second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1hour, 2 hours, 5 hours, or 10 hours.
 11. The method according to claim9, wherein the time-interval is more than 1 millisecond, 2 milliseconds,5 milliseconds, 10 milliseconds, 20 milliseconds, 30 milliseconds, 50milliseconds, 100 milliseconds, 200 milliseconds, 500 milliseconds, 1second, 2 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1minute, 2, minutes, 5 minutes, 10 minutes, 20 minutes, 30 minutes, 1hour, 2 hours, 5 hours, or 10 hours.
 12. The method according to claim1, wherein the comparing comprises, is based on, or uses, convolution,correlation, or a cross-correlation operation on the first and secondsignals.
 13. The method according to claim 12, wherein the comparingcomprises, consists of, is based on, or uses, calculating or estimatinga cross-correlation coefficient of the first and second signals.
 14. Themethod according to claim 12, wherein the estimating of the time delayvalue comprises, consists of, is based on, or uses, selecting a timedelay value that results in a maximum value of a cross-correlationcoefficient value when time shifting the first or second signal by thetime delay value.
 15. The method according to claim 12, wherein thecomparing comprises, consists of, is based on, or uses,Cross-Correlation (CC), Phase Transform (PHAT), Maximum Likelihoodestimator (ML), Adaptive Least Mean Square filter (LMS), or AverageSquare Difference Function (ASDF).
 16. The method according to claim 1,for use with a third signal that is based on, is a function of, or inresponse to the first signal, and for use with a fourth signal that isbased on, is a function of, or in response to the second signal, whereinthe comparing comprises detecting or identifying a first event in thethird signal at a first time point and detecting or identifying a secondevent in the fourth signal at a second time point, and wherein the timedelay value is estimated based on, equal to, or a function of, of a timedifferent between the first and second time points.
 17. The methodaccording to claim 16, wherein the first event or the second eventcomprises detecting or identifying a peak value in the respective thirdor fourth signal.
 18. The method according to claim 16, for use with athreshold value, wherein the first event or the second event comprisesdetecting or identifying a threshold value crossing in the respectivethird or fourth signal.
 19. The method according to claim 16, whereinthe third signal is the first signal or the fourth signal is the secondsignal.
 20. The method according to claim 16, further comprisingproducing the third signal by applying a time-domain analysis ormanipulation to the first signal and producing the fourth signal byapplying a time-domain analysis or manipulation to the second signal.21. The method according to claim 20, wherein the time-domain analysisor manipulation comprises detecting zero crossings, peak amplitude,rise-time, or energy.
 22. The method according to claim 20, wherein thetime-domain analysis or manipulation comprises a Mel-Frequency Analysis,calculating Mel-Frequency Cepstral Coefficients (MFCC), using a LinearPredictive Coding (LPC), or calculating LPC coefficients.
 23. The methodaccording to claim 20, wherein the time-domain analysis or manipulationcomprises a discrete, continuous, monotonic, non-monotonic, elementary,algebraic, linear, polynomial, quadratic, Cubic, Nth-root based,exponential, transcendental, quintic, quartic, logarithmic, hyperbolic,or trigonometric function.
 24. The method according to claim 16, furthercomprising producing the third signal by applying a frequency-domainanalysis or manipulation to the first signal and producing the fourthsignal by applying a frequency-domain analysis or manipulation to thesecond signal.
 25. The method according to claim 24, wherein thefrequency-domain analysis or manipulation comprises Fourier series,Fourier transform, Laplace transform, Z transform, or Wavelet transform.26. The method according to claim 1, wherein the digital video cameracomprises: an optical lens for focusing received light, the lens beingmechanically oriented to guide a captured image; a photosensitive imagesensor array disposed approximately at an image focal point plane of theoptical lens for capturing the image and producing an analog signalrepresenting the image; and an analog-to-digital (A/D) converter coupledto the image sensor array for converting the analog signal to the videodata stream.
 27. The method according to claim 26, wherein the imagesensor array is operative to respond to a visible or non-visible light.28. The method according to claim 27, wherein the invisible light isinfrared, ultraviolet, X-rays, or gamma rays.
 29. An apparatuscomprises: a memory or a non-transitory tangible computer readablestorage media for storing computer executable instructions to performthe method of claim 1; and a processor for executing the instructions.30. A non-transitory computer readable medium having computer executableinstructions stored thereon, wherein the instructions include the stepsof claim 1.